mirror of
https://github.com/JHUAPL/AccumuloGraph.git
synced 2026-01-08 20:28:03 -05:00
Compare commits
196 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f3f8115c91 | ||
|
|
91623277fb | ||
|
|
8b0a131f5c | ||
|
|
23df037c13 | ||
|
|
d954f1b4bd | ||
|
|
033d29320c | ||
|
|
a9813d0236 | ||
|
|
46112bbc32 | ||
|
|
c5a42df3aa | ||
|
|
9b66771627 | ||
|
|
05c6067303 | ||
|
|
b1a9a88fd6 | ||
|
|
e6464b59e3 | ||
|
|
13034977ea | ||
|
|
59b49f01ba | ||
|
|
1f630eb3ef | ||
|
|
718258e828 | ||
|
|
99e01d8608 | ||
|
|
a42ae0a521 | ||
|
|
00056f6513 | ||
|
|
8df511c2d1 | ||
|
|
ec13b05459 | ||
|
|
37157a0093 | ||
|
|
438830f5f4 | ||
|
|
1018d48382 | ||
|
|
30468ed2f3 | ||
|
|
6faac76ba9 | ||
|
|
bb6f9e80a7 | ||
|
|
56929af885 | ||
|
|
89976c2c86 | ||
|
|
816dffe141 | ||
|
|
a7035e259a | ||
|
|
f745ff23fb | ||
|
|
93c9c37936 | ||
|
|
c08bb5058f | ||
|
|
dcea092c40 | ||
|
|
f9bb666e4b | ||
|
|
834e2cb2aa | ||
|
|
f72b1d2f2f | ||
|
|
5dd5c978a7 | ||
|
|
bca5bf4474 | ||
|
|
365dc740d7 | ||
|
|
d5de29c000 | ||
|
|
0b2ec50eeb | ||
|
|
8a36bc7930 | ||
|
|
38b22df42f | ||
|
|
1123268b37 | ||
|
|
be81489672 | ||
|
|
37774d60a1 | ||
|
|
e2c5af3752 | ||
|
|
9afc558d95 | ||
|
|
a2502f5bb1 | ||
|
|
49a5a77d1d | ||
|
|
53f7ed4025 | ||
|
|
9edf2ff66e | ||
|
|
49d4463cc5 | ||
|
|
37593c13b4 | ||
|
|
7bf543e502 | ||
|
|
43a3293ef0 | ||
|
|
6fffd094f3 | ||
|
|
b60f855a5e | ||
|
|
832d597f0a | ||
|
|
7fb9cce693 | ||
|
|
128b40346a | ||
|
|
eacf44c755 | ||
|
|
e59c590eae | ||
|
|
2dd9e66cf2 | ||
|
|
3897998fca | ||
|
|
a9d8ff7570 | ||
|
|
7e79174b1d | ||
|
|
df0824f7de | ||
|
|
864cee9312 | ||
|
|
2544ac0c30 | ||
|
|
16b7284f08 | ||
|
|
f0bd6b8cd6 | ||
|
|
379099adc2 | ||
|
|
f0a23e3848 | ||
|
|
ad6ef0c2f1 | ||
|
|
5632417f30 | ||
|
|
c17693968c | ||
|
|
c66d86d123 | ||
|
|
36ca8641ef | ||
|
|
5032b72803 | ||
|
|
c90a379e2b | ||
|
|
9832b92d7b | ||
|
|
712420c2c5 | ||
|
|
2c08dc149e | ||
|
|
484e78f080 | ||
|
|
2b3f07688d | ||
|
|
ff17e6a5af | ||
|
|
72c9685450 | ||
|
|
42afab5968 | ||
|
|
332df1fea4 | ||
|
|
6196dc7d8f | ||
|
|
98a24bd62f | ||
|
|
c9b7473df0 | ||
|
|
886c394afb | ||
|
|
8b7f8b7e9d | ||
|
|
25eb8e9979 | ||
|
|
33042acedc | ||
|
|
2465f65482 | ||
|
|
899ed892fc | ||
|
|
ad7d405339 | ||
|
|
e68f396dfe | ||
|
|
8afc46ce8b | ||
|
|
7c8996ba8b | ||
|
|
37b4c8adc9 | ||
|
|
76b0007434 | ||
|
|
3a08194ba3 | ||
|
|
65b8d43d42 | ||
|
|
8b8fc0c7e6 | ||
|
|
fec6fdc522 | ||
|
|
439dc7ae93 | ||
|
|
ac68e334a6 | ||
|
|
f12e78d836 | ||
|
|
2d7a0d4c22 | ||
|
|
06bb292b42 | ||
|
|
6b85e54628 | ||
|
|
ed77c11b1c | ||
|
|
a61fc8ba20 | ||
|
|
6613bcd569 | ||
|
|
80ddbd3127 | ||
|
|
c1429b3bf3 | ||
|
|
aa860b3441 | ||
|
|
1c27add8e5 | ||
|
|
9c2e07c395 | ||
|
|
bcc4af3599 | ||
|
|
85544dea77 | ||
|
|
75df33d51c | ||
|
|
e126a52c32 | ||
|
|
f0d3898c2d | ||
|
|
ee45dc4ade | ||
|
|
f90290bb3d | ||
|
|
8c6c02dffb | ||
|
|
e2dccb15c2 | ||
|
|
7a829d5c34 | ||
|
|
0f854ec2b7 | ||
|
|
99c19669f9 | ||
|
|
f94bb2e9f4 | ||
|
|
ef978884bc | ||
|
|
7ffd181dbe | ||
|
|
2ffdb81fe3 | ||
|
|
c29983966b | ||
|
|
bfba0ed665 | ||
|
|
a3181eac62 | ||
|
|
7d2d9bb45a | ||
|
|
bed037caa2 | ||
|
|
be75b8fc1f | ||
|
|
fc09015445 | ||
|
|
805368972f | ||
|
|
cae1124562 | ||
|
|
101c549f76 | ||
|
|
fc827aa0ca | ||
|
|
aa4897ca27 | ||
|
|
8b120f66ed | ||
|
|
9117cab1ae | ||
|
|
0526fb68e4 | ||
|
|
5affcea6e7 | ||
|
|
1acc82af57 | ||
|
|
1590fe05fd | ||
|
|
33cc53c00e | ||
|
|
0b8ab3361a | ||
|
|
b991d84a28 | ||
|
|
bd7cd409d2 | ||
|
|
b7d9d3f340 | ||
|
|
0bf11f444e | ||
|
|
5d7442e024 | ||
|
|
670e4b5c85 | ||
|
|
2db2960b67 | ||
|
|
f378931c3a | ||
|
|
b749943cc9 | ||
|
|
d20da99ad3 | ||
|
|
6eba6cc502 | ||
|
|
b5d5c6f5bc | ||
|
|
2a74321b5a | ||
|
|
1b58eb42e8 | ||
|
|
100c46aee9 | ||
|
|
f2406b8f07 | ||
|
|
7f8c98edc6 | ||
|
|
1309512331 | ||
|
|
c42c06f6e2 | ||
|
|
dd98652d92 | ||
|
|
de42c76fa5 | ||
|
|
e1d359ab83 | ||
|
|
b52b441dfd | ||
|
|
db35a8f620 | ||
|
|
0d9eba2211 | ||
|
|
955177b46e | ||
|
|
d699dbee8f | ||
|
|
60e455a4ba | ||
|
|
289f65999d | ||
|
|
507ffa180c | ||
|
|
1b10773c6c | ||
|
|
83e227e760 | ||
|
|
0b23289d07 | ||
|
|
abd1771803 |
308
README.md
308
README.md
@@ -4,140 +4,230 @@ AccumuloGraph
|
||||
|
||||
This is an implementation of the [TinkerPop Blueprints](http://tinkerpop.com)
|
||||
2.6 API using [Apache Accumulo](http://apache.accumulo.com) as the backend.
|
||||
This implementation provides easy to use, easy to write, and easy to read
|
||||
access to an arbitrarily large graph that is stored in Accumulo.
|
||||
|
||||
We implement the following Blueprints interfaces:
|
||||
<br>1. Graph
|
||||
<br>2. KeyIndexableGraph
|
||||
<br>3. IndexableGraph
|
||||
|
||||
Please feel free to submit issues for any bugs you find or features you want.
|
||||
We are open to pull requests from your forks also.
|
||||
This combines the many benefits and flexibility of Blueprints
|
||||
with the scalability and performance of Accumulo.
|
||||
|
||||
##Usage
|
||||
In addition to the basic Blueprints functionality, we provide a number
|
||||
of enhanced features, including:
|
||||
* Indexing implementations via `IndexableGraph` and `KeyIndexableGraph`
|
||||
* Support for mock, mini, and distributed instances of Accumulo
|
||||
* Numerous performance tweaks and configuration parameters
|
||||
* Support for high speed ingest
|
||||
* Hadoop integration
|
||||
|
||||
Feel free to contact us with bugs, suggestions, pull requests,
|
||||
or simply how you are leveraging AccumuloGraph in your own work.
|
||||
|
||||
|
||||
## Getting Started
|
||||
|
||||
First, include AccumuloGraph as a Maven dependency. Releases are deployed
|
||||
to Maven Central.
|
||||
|
||||
The releases are currently stored in Maven Central.
|
||||
```xml
|
||||
<dependency>
|
||||
<groupId>edu.jhuapl.tinkerpop</groupId>
|
||||
<artifactId>blueprints-accumulo-graph</artifactId>
|
||||
<version>0.0.1</version>
|
||||
<version>0.2.1</version>
|
||||
</dependency>
|
||||
```
|
||||
|
||||
For non-Maven users, the binaries can be found in the releases section in this
|
||||
For non-Maven users, the binary jars can be found in the releases section in this
|
||||
GitHub repository, or you can get them from Maven Central.
|
||||
##Code Examples
|
||||
###Creating a new or connecting to an existing distributed graph
|
||||
|
||||
Creating an `AccumuloGraph` involves setting a few parameters in an
|
||||
`AccumuloGraphConfiguration` object, and opening the graph.
|
||||
The defaults are sensible for using an Accumulo cluster.
|
||||
We provide some simple examples below. Javadocs for
|
||||
`AccumuloGraphConfiguration` explain all the other parameters
|
||||
in more detail.
|
||||
|
||||
First, to instantiate an in-memory graph:
|
||||
```java
|
||||
Configuration cfg = new AccumuloGraphConfiguration()
|
||||
.setInstanceName("accumulo").setUser("user").setZookeeperHosts("zk1")
|
||||
.setPassword("password".getBytes()).setGraphName("myGraph");
|
||||
Graph graph = GraphFactory.open(cfg.getConfiguration());
|
||||
.setInstanceType(InstanceType.Mock)
|
||||
.setGraphName("graph");
|
||||
return GraphFactory.open(cfg);
|
||||
```
|
||||
###Creating a new Mock Graph
|
||||
|
||||
Setting the instance type to mock allows for in-memory processing with a MockAccumulo instance.<br>
|
||||
There is also support for Mini Accumulo.
|
||||
This creates a "Mock" instance which holds the graph in memory.
|
||||
You can now use all the Blueprints and AccumuloGraph-specific functionality
|
||||
with this in-memory graph. This is useful for getting familiar
|
||||
with AccumuloGraph's functionality, or for testing or prototyping
|
||||
purposes.
|
||||
|
||||
To use an actual Accumulo cluster, use the following:
|
||||
```java
|
||||
Configuration cfg = new AccumuloGraphConfiguration().setInstanceType(InstanceType.Mock)
|
||||
.setGraphName("myGraph");
|
||||
Graph graph = GraphFactory.open(cfg);
|
||||
Configuration cfg = new AccumuloGraphConfiguration()
|
||||
.setInstanceType(InstanceType.Distributed)
|
||||
.setZooKeeperHosts("zookeeper-host")
|
||||
.setInstanceName("instance-name")
|
||||
.setUser("user").setPassword("password")
|
||||
.setGraphName("graph")
|
||||
.setCreate(true);
|
||||
return GraphFactory.open(cfg);
|
||||
```
|
||||
###Accessing a graph
|
||||
|
||||
This directs AccumuloGraph to use a "Distributed" Accumulo
|
||||
instance, and sets the appropriate ZooKeeper parameters,
|
||||
instance name, and authentication information, which correspond
|
||||
to the usual Accumulo connection settings. The graph name is
|
||||
used to create several backing tables in Accumulo, and the
|
||||
`setCreate` option tells AccumuloGraph to create the backing
|
||||
tables if they don't already exist.
|
||||
|
||||
AccumuloGraph also has limited support for a "Mini" instance
|
||||
of Accumulo.
|
||||
|
||||
|
||||
## Improving Performance
|
||||
|
||||
This section describes various configuration parameters that
|
||||
greatly enhance AccumuloGraph's performance. Brief descriptions
|
||||
of each option are provided here, but refer to the
|
||||
`AccumuloGraphConfiguration` Javadoc for fuller explanations.
|
||||
|
||||
### Disable consistency checks
|
||||
|
||||
The Blueprints API specifies a number of consistency checks for
|
||||
various operations, and requires errors if they fail. Some examples
|
||||
of invalid operations include adding a vertex with the same id as an
|
||||
existing vertex, adding edges between nonexistent vertices,
|
||||
and setting properties on nonexistent elements.
|
||||
Unfortunately, checking the above constraints for an
|
||||
Accumulo installation entails significant performance issues,
|
||||
since these require extra traffic to Accumulo using inefficient
|
||||
non-batched access patterns.
|
||||
|
||||
To remedy these performance issues, AccumuloGraph exposes
|
||||
several options to disable various of the above checks.
|
||||
These include:
|
||||
* `setAutoFlush` - to disable automatically flushing
|
||||
changes to the backing Accumulo tables
|
||||
* `setSkipExistenceChecks` - to disable element
|
||||
existence checks, avoiding trips to the Accumulo cluster
|
||||
* `setIndexableGraphDisabled` - to disable
|
||||
indexing functionality, which improves performance
|
||||
of element removal
|
||||
|
||||
### Tweak Accumulo performance parameters
|
||||
|
||||
Accumulo itself features a number of performance-related parameters,
|
||||
and we allow configuration of these. Generally, these relate to
|
||||
write buffer sizes, multithreading, etc. The settings include:
|
||||
* `setMaxWriteLatency` - max time prior to flushing
|
||||
element write buffer
|
||||
* `setMaxWriteMemory` - max size for element write buffer
|
||||
* `setMaxWriteThreads` - max threads used for element writing
|
||||
* `setMaxWriteTimeout` - max time to wait before failing
|
||||
element buffer writes
|
||||
* `setQueryThreads` - number of query threads to use
|
||||
for fetching elements, properties etc.
|
||||
|
||||
### Enable edge and property preloading
|
||||
|
||||
As a performance tweak, AccumuloGraph performs lazy loading of
|
||||
properties and edges. This means that an operation such as
|
||||
`getVertex` does not by default populate the returned
|
||||
vertex object with the associated vertex's properties
|
||||
and edges. Instead, they are initialized only when requested via
|
||||
`getProperty`, `getEdges`, etc. These are useful
|
||||
for use cases where you won't be accessing many of these
|
||||
properties. However, if certain properties or edges will
|
||||
be accessed frequently, you can set options for preloading
|
||||
these specific properties and edges, which will be more
|
||||
efficient than on-the-fly loading. These options include:
|
||||
* `setPreloadedProperties` - set property keys
|
||||
to be preloaded
|
||||
* `setPreloadedEdgeLabels` - set edges to be
|
||||
preloaded based on their labels
|
||||
|
||||
### Enable caching
|
||||
|
||||
AccumuloGraph contains a number of caching options
|
||||
that mitigate the need for Accumulo traffic for recently-accessed
|
||||
elements. The following options control caching:
|
||||
* `setVertexCacheParams` - size and expiry for vertex cache
|
||||
* `setEdgeCacheParams` - size and expiry for edge cache
|
||||
* `setPropertyCacheTimeout` - property expiry time,
|
||||
which can be specified globally and/or for individual properties
|
||||
|
||||
|
||||
## High Speed Ingest
|
||||
|
||||
One of Accumulo's key advantages is its ability for high-speed ingest
|
||||
of huge amounts of data. To leverage this ability, we provide
|
||||
an additional `AccumuloBulkIngester` class that
|
||||
exchanges consistency guarantees for high speed ingest.
|
||||
|
||||
The following is an example of how to use the bulk ingester to
|
||||
ingest a simple graph:
|
||||
```java
|
||||
Vertex v1 = graph.addVertex("1");
|
||||
v1.setProperty("name", "Alice");
|
||||
Vertex v2 = graph.addVertex("2");
|
||||
v2.setProperty("name", "Bob");
|
||||
|
||||
Edge e1 = graph.addEdge("E1", v1, v2, "knows");
|
||||
e1.setProperty("since", new Date());
|
||||
```
|
||||
|
||||
|
||||
###Creating indexes
|
||||
|
||||
```java
|
||||
((KeyIndexableGraph)graph)
|
||||
.createKeyIndex("name", Vertex.class);
|
||||
AccumuloGraphConfiguration cfg = ...;
|
||||
AccumuloBulkIngester ingester = new AccumuloBulkIngester(cfg);
|
||||
// Add a vertex.
|
||||
ingester.addVertex("A").finish();
|
||||
// Add another vertex with properties.
|
||||
ingester.addVertex("B")
|
||||
.add("P1", "V1").add("P2", "V2")
|
||||
.finish();
|
||||
// Add an edge.
|
||||
ingester.addEdge("A", "B", "edge").finish();
|
||||
// Shutdown and compact tables.
|
||||
ingester.shutdown(true);
|
||||
```
|
||||
###MapReduce Integration
|
||||
|
||||
####In the tool
|
||||
See the Javadocs for more details.
|
||||
Note that you are responsible for ensuring that data is entered
|
||||
in a consistent way, or the resulting graph will
|
||||
have undefined behavior.
|
||||
|
||||
|
||||
## Hadoop Integration
|
||||
|
||||
AccumuloGraph features Hadoop integration via custom input and output
|
||||
format implementations. `VertexInputFormat` and `EdgeInputFormat`
|
||||
allow vertex and edge inputs to mappers, respectively. Use as follows:
|
||||
```java
|
||||
AccumuloConfiguration cfg = new AccumuloGraphConfiguration()
|
||||
.setInstanceName("accumulo").setZookeeperHosts("zk1").setUser("root")
|
||||
.setPassword("secret".getBytes()).setGraphName("myGraph");
|
||||
AccumuloGraphConfiguration cfg = ...;
|
||||
|
||||
// For vertices:
|
||||
Job j = new Job();
|
||||
j.setInputFormatClass(VertexInputFormat.class);
|
||||
VertexInputFormat.setAccumuloGraphConfiguration(j,
|
||||
cfg.getConfiguration());
|
||||
VertexInputFormat.setAccumuloGraphConfiguration(j, cfg);
|
||||
|
||||
// For edges:
|
||||
Job j = new Job();
|
||||
j.setInputFormatClass(EdgeInputFormat.class);
|
||||
EdgeInputFormat.setAccumuloGraphConfiguration(j, cfg);
|
||||
```
|
||||
####In the mapper
|
||||
|
||||
`ElementOutputFormat` allows writing to an AccumuloGraph from
|
||||
reducers. Use as follows:
|
||||
```java
|
||||
public void map(Text k, Vertex v, Context c) {
|
||||
System.out.println(v.getId().toString());
|
||||
}
|
||||
```
|
||||
AccumuloGraphConfiguration cfg = ...;
|
||||
|
||||
##Table Design
|
||||
###Vertex Table
|
||||
Row ID | Column Family | Column Qualifier | Value
|
||||
---|---|---|---
|
||||
VertexID | Label Flag | Exists Flag | [empty]
|
||||
VertexID | INVERTEX | OutVertexID_EdgeID | Edge Label
|
||||
VertexID | OUTVERTEX | InVertexID_EdgeID | Edge Label
|
||||
VertexID | Property Key | [empty] | Serialized Value
|
||||
###Edge Table
|
||||
Row ID | Column Family | Column Qualifier | Value
|
||||
---|---|---|---
|
||||
EdgeID|Label Flag|InVertexID_OutVertexID|Edge Label
|
||||
EdgeID|Property Key|[empty]|Serialized Value
|
||||
###Edge/Vertex Index
|
||||
Row ID | Column Family | Column Qualifier | Value
|
||||
---|---|---|---
|
||||
Serialized Value|Property Key|VertexID/EdgeID|[empty]
|
||||
Job j = new Job();
|
||||
j.setOutputFormatClass(ElementOutputFormat.class);
|
||||
ElementOutputFormat.setAccumuloGraphConfiguration(j, cfg);
|
||||
```
|
||||
|
||||
###Metadata Table
|
||||
Row ID | Column Family | Column Qualifier | Value
|
||||
---|---|---|---
|
||||
Index Name| Index Class |[empty]|[empty]
|
||||
##Advanced Configuration
|
||||
###Graph Configuration
|
||||
- setGraphName(String name)
|
||||
- setCreate(boolean create) - Sets if the backing graph tables should be created if they do not exist.
|
||||
- setClear(boolean clear) - Sets if the backing graph tables should be reset if they exist.
|
||||
- autoFlush(boolean autoFlush) - Sets if each graph element and property change will be flushed to the server.
|
||||
- skipExistenceChecks(boolean skip) - Sets if you want to skip existance checks when creating graph elemenets.
|
||||
- setAutoIndex(boolean ison) - Turns on/off automatic indexing.
|
||||
|
||||
###Accumulo Control
|
||||
|
||||
- setUser(String user) - Sets the user to use when connecting to Accumulo
|
||||
- setPassword(byte[] password | String password) - Sets the password to use when connecting to Accumulo
|
||||
- setZookeeperHosts(String zookeeperHosts) - Sets the Zookeepers to connect to.
|
||||
- setInstanceName(String instance) - Sets the Instance name to use when connecting to Zookeeper
|
||||
- setInstanceType(InstanceType type) - Sets the type of Instance to use : Distrubuted, Mini, or Mock. Defaults to Distrubuted
|
||||
- setQueryThreads(int threads) - Specifies the number of threads to use in scanners. Defaults to 3
|
||||
- setMaxWriteLatency(long latency) - Sets the latency to be used for all writes to Accumulo
|
||||
- setMaxWriteTimeout(long timeout) - Sets the timeout to be used for all writes to Accumulo
|
||||
- setMaxWriteMemory(long mem) - Sets the memory buffer to be used for all writes to Accumulo
|
||||
- setMaxWriteThreads(int threads) - Sets the number of threads to be used for all writes to Accumulo
|
||||
- setAuthorizations(Authorizations auths) - Sets the authorizations to use when accessing the graph
|
||||
- setColumnVisibility(ColumnVisibility colVis) - TODO
|
||||
- setSplits(String splits | String[] splits) - Sets the splits to use when creating tables. Can be a space sperated list or an array of splits
|
||||
- setMiniClusterTempDir(String dir) - Sets directory to use as the temp directory for the Mini cluster
|
||||
|
||||
###Caching
|
||||
- setLruMaxCapacity(int max) - TODO
|
||||
- setVertexCacheTimeout(int millis) - Sets the vertex cache timeout. A value <=0 clears the value
|
||||
- setEdgeCacheTimeout(int millis) - Sets the edge cache timeout. A value <=0 clears the value
|
||||
|
||||
###Preloading
|
||||
- setPropertyCacheTimeout(int millis) - Sets the element property cache timeout. A value <=0 clears the value
|
||||
- setPreloadedProperties(String[] propertyKeys) - Sets the property keys that should be preloaded. Requiers a positive timout.
|
||||
- setPreloadedEdgeLabels(String[] edgeLabels) - TODO
|
||||
## Rexster Configuration
|
||||
Below is a snippet to show an example of AccumuloGraph integration with Rexster. For a complete list of options for configuration, see [`AccumuloGraphConfiguration$Keys`](https://github.com/JHUAPL/AccumuloGraph/blob/master/src/main/java/edu/jhuapl/tinkerpop/AccumuloGraphConfiguration.java#L110)
|
||||
|
||||
```xml
|
||||
<graph>
|
||||
<graph-enabled>true</graph-enabled>
|
||||
<graph-name>myGraph</graph-name>
|
||||
<graph-type>edu.jhuapl.tinkerpop.AccumuloRexsterGraphConfiguration</graph-type>
|
||||
<properties>
|
||||
<blueprints.accumulo.instance.type>Distributed</blueprints.accumulo.instance.type>
|
||||
<blueprints.accumulo.instance>accumulo</blueprints.accumulo.instance>
|
||||
<blueprints.accumulo.zkhosts>zk1,zk2,zk3</blueprints.accumulo.zkhosts>
|
||||
<blueprints.accumulo.user>user</blueprints.accumulo.user>
|
||||
<blueprints.accumulo.password>password</blueprints.accumulo.password>
|
||||
</properties>
|
||||
<extensions>
|
||||
</extensions>
|
||||
</graph>
|
||||
```
|
||||
|
||||
10
changelog
Normal file
10
changelog
Normal file
@@ -0,0 +1,10 @@
|
||||
[2.2 Snapshot]
|
||||
-- 113 Applied instance name when mock instance is used
|
||||
-- 114 Made map reduce elements serializable
|
||||
-- 116 Made MapReduceElement serializable with a tranisent graph
|
||||
[2.1]
|
||||
Change Log started
|
||||
--109,103 Merged Metadata tables
|
||||
--97 Added range queries
|
||||
--95 Table structure documentation
|
||||
--79 All flag for preloading
|
||||
86
pom.xml
86
pom.xml
@@ -3,7 +3,7 @@
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
<groupId>edu.jhuapl.tinkerpop</groupId>
|
||||
<artifactId>blueprints-accumulo-graph</artifactId>
|
||||
<version>0.0.2</version>
|
||||
<version>0.2.2-SNAPSHOT</version>
|
||||
<name>blueprints-accumulo-graph</name>
|
||||
<description>An Accumulo-backed implementation of the Tinkerpop Blueprints graph API.</description>
|
||||
<packaging>jar</packaging>
|
||||
@@ -14,8 +14,6 @@
|
||||
<url>http://www.apache.org/licenses/LICENSE-2.0.txt</url>
|
||||
<distribution>repo</distribution>
|
||||
</license>
|
||||
|
||||
|
||||
</licenses>
|
||||
|
||||
<scm>
|
||||
@@ -43,7 +41,6 @@
|
||||
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
|
||||
</properties>
|
||||
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.apache.accumulo</groupId>
|
||||
@@ -68,7 +65,19 @@
|
||||
<dependency>
|
||||
<groupId>commons-collections</groupId>
|
||||
<artifactId>commons-collections</artifactId>
|
||||
<version>3.2.1</version>
|
||||
<version>3.2.2</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.apache.hadoop</groupId>
|
||||
<artifactId>hadoop-client</artifactId>
|
||||
<version>2.2.0</version>
|
||||
<scope>provided</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.tinkerpop.rexster</groupId>
|
||||
<artifactId>rexster-server</artifactId>
|
||||
<version>2.6.0</version>
|
||||
<scope>provided</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.tinkerpop.blueprints</groupId>
|
||||
@@ -76,18 +85,12 @@
|
||||
<version>2.6.0</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.apache.hadoop</groupId>
|
||||
<artifactId>hadoop-client</artifactId>
|
||||
<version>2.2.0</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.apache.hadoop</groupId>
|
||||
<artifactId>hadoop-minicluster</artifactId>
|
||||
<version>2.2.0</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>junit</groupId>
|
||||
<artifactId>junit</artifactId>
|
||||
@@ -100,12 +103,6 @@
|
||||
<version>3.1</version>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.tinkerpop.rexster</groupId>
|
||||
<artifactId>rexster-server</artifactId>
|
||||
<version>2.6.0</version>
|
||||
</dependency>
|
||||
|
||||
</dependencies>
|
||||
|
||||
<distributionManagement>
|
||||
@@ -167,6 +164,59 @@
|
||||
</plugins>
|
||||
</build>
|
||||
</profile>
|
||||
<profile>
|
||||
<id>javadoc</id>
|
||||
<build>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-source-plugin</artifactId>
|
||||
<version>2.2.1</version>
|
||||
<executions>
|
||||
<execution>
|
||||
<id>attach-sources</id>
|
||||
<goals>
|
||||
<goal>jar-no-fork</goal>
|
||||
</goals>
|
||||
</execution>
|
||||
</executions>
|
||||
</plugin>
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-javadoc-plugin</artifactId>
|
||||
<version>2.9.1</version>
|
||||
<executions>
|
||||
<execution>
|
||||
<id>attach-javadocs</id>
|
||||
<goals>
|
||||
<goal>jar</goal>
|
||||
</goals>
|
||||
</execution>
|
||||
</executions>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
</profile>
|
||||
<profile>
|
||||
<id>fatjar</id>
|
||||
<build>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-shade-plugin</artifactId>
|
||||
<version>2.3</version>
|
||||
<executions>
|
||||
<execution>
|
||||
<phase>package</phase>
|
||||
<goals>
|
||||
<goal>shade</goal>
|
||||
</goals>
|
||||
</execution>
|
||||
</executions>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
</profile>
|
||||
</profiles>
|
||||
|
||||
<build>
|
||||
@@ -182,8 +232,6 @@
|
||||
<autoReleaseAfterClose>true</autoReleaseAfterClose>
|
||||
</configuration>
|
||||
</plugin>
|
||||
|
||||
</plugins>
|
||||
</build>
|
||||
|
||||
</project>
|
||||
|
||||
@@ -32,11 +32,20 @@ import com.tinkerpop.blueprints.Edge;
|
||||
import com.tinkerpop.blueprints.GraphFactory;
|
||||
import com.tinkerpop.blueprints.Vertex;
|
||||
|
||||
import edu.jhuapl.tinkerpop.mutator.Mutators;
|
||||
import edu.jhuapl.tinkerpop.mutator.edge.EdgeEndpointsMutator;
|
||||
import edu.jhuapl.tinkerpop.mutator.edge.EdgeMutator;
|
||||
import edu.jhuapl.tinkerpop.mutator.property.WritePropertyMutator;
|
||||
import edu.jhuapl.tinkerpop.mutator.vertex.AddVertexMutator;
|
||||
|
||||
|
||||
/**
|
||||
* This class providers high-speed ingest into an AccumuloGraph instance in exchange for consistency guarantees. That is, users of this class must ensure
|
||||
* (outside of this class) that data is entered in a consistent way or the behavior or the resulting AccumuloGraph is undefined. For example, users are required
|
||||
* to ensure that a vertex ID provided as the source or destination of an edge exists (or will exist by the end of the ingest process). Likewise, it is the
|
||||
* This class provides high-speed ingest into an {@link AccumuloGraph} instance
|
||||
* in exchange for consistency guarantees. That is, users of this class must ensure
|
||||
* (outside of this class) that data is entered in a consistent way or the behavior
|
||||
* or the resulting AccumuloGraph is undefined. For example, users are required
|
||||
* to ensure that a vertex ID provided as the source or destination of an edge exists
|
||||
* (or will exist by the end of the ingest process). Likewise, it is the
|
||||
* user's responsibility to ensure vertex and edge IDs provided for properties (will) exist.
|
||||
* <P>
|
||||
* TODO define the properties that will be used (vs. those that are ignored) from the provided AccumuloGraphConfiguration.
|
||||
@@ -88,8 +97,8 @@ public final class AccumuloBulkIngester {
|
||||
AccumuloGraphUtils.handleCreateAndClear(config);
|
||||
|
||||
mtbw = connector.createMultiTableBatchWriter(config.getBatchWriterConfig());
|
||||
vertexWriter = mtbw.getBatchWriter(config.getVertexTable());
|
||||
edgeWriter = mtbw.getBatchWriter(config.getEdgeTable());
|
||||
vertexWriter = mtbw.getBatchWriter(config.getVertexTableName());
|
||||
edgeWriter = mtbw.getBatchWriter(config.getEdgeTableName());
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -105,9 +114,7 @@ public final class AccumuloBulkIngester {
|
||||
* @throws MutationsRejectedException
|
||||
*/
|
||||
public PropertyBuilder addVertex(String id) throws MutationsRejectedException {
|
||||
Mutation m = new Mutation(id);
|
||||
m.put(AccumuloGraph.LABEL, AccumuloGraph.EXISTS, AccumuloGraph.EMPTY);
|
||||
vertexWriter.addMutation(m);
|
||||
Mutators.apply(vertexWriter, new AddVertexMutator(id));
|
||||
return new PropertyBuilder(vertexWriter, id);
|
||||
}
|
||||
|
||||
@@ -145,8 +152,7 @@ public final class AccumuloBulkIngester {
|
||||
* @throws MutationsRejectedException
|
||||
*/
|
||||
public PropertyBuilder addEdge(String src, String dest, String label) throws MutationsRejectedException {
|
||||
String eid = UUID.randomUUID().toString();
|
||||
return addEdge(eid, src, dest, label);
|
||||
return addEdge(UUID.randomUUID().toString(), src, dest, label);
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -165,16 +171,8 @@ public final class AccumuloBulkIngester {
|
||||
* @throws MutationsRejectedException
|
||||
*/
|
||||
public PropertyBuilder addEdge(String id, String src, String dest, String label) throws MutationsRejectedException {
|
||||
Mutation m = new Mutation(id);
|
||||
m.put(AccumuloGraph.LABEL, (dest + "_" + src).getBytes(), AccumuloByteSerializer.serialize(label));
|
||||
edgeWriter.addMutation(m);
|
||||
|
||||
m = new Mutation(dest);
|
||||
m.put(AccumuloGraph.INEDGE, (src + AccumuloGraph.IDDELIM + id).getBytes(), (AccumuloGraph.IDDELIM + label).getBytes());
|
||||
vertexWriter.addMutation(m);
|
||||
m = new Mutation(src);
|
||||
m.put(AccumuloGraph.OUTEDGE, (dest + AccumuloGraph.IDDELIM + id).getBytes(), (AccumuloGraph.IDDELIM + label).getBytes());
|
||||
vertexWriter.addMutation(m);
|
||||
Mutators.apply(edgeWriter, new EdgeMutator.Add(id, src, dest, label));
|
||||
Mutators.apply(vertexWriter, new EdgeEndpointsMutator.Add(id, src, dest, label));
|
||||
return new PropertyBuilder(edgeWriter, id);
|
||||
}
|
||||
|
||||
@@ -207,22 +205,27 @@ public final class AccumuloBulkIngester {
|
||||
* @throws MutationsRejectedException
|
||||
*/
|
||||
private void addProperty(BatchWriter writer, String id, String key, Object value) throws MutationsRejectedException {
|
||||
byte[] newByteVal = AccumuloByteSerializer.serialize(value);
|
||||
Mutation m = new Mutation(id);
|
||||
m.put(key.getBytes(), AccumuloGraph.EMPTY, newByteVal);
|
||||
writer.addMutation(m);
|
||||
Mutators.apply(writer, new WritePropertyMutator(id, key, value));
|
||||
}
|
||||
|
||||
/**
|
||||
* Shutdown the bulk ingester. This flushes any outstanding writes to Accumulo and performs any remaining clean up to finalize the graph.
|
||||
* Shutdown the bulk ingester. This flushes any outstanding writes to Accumulo
|
||||
* and performs any remaining clean up to finalize the graph.
|
||||
*
|
||||
* @param compact
|
||||
* a flag if this shutdown should kick off a compaction on the graph-related tables (true) or not (false) before quiting.
|
||||
* a flag if this shutdown should start a compaction on the graph-related tables
|
||||
* (true) or not (false) before quitting.
|
||||
* @throws AccumuloException
|
||||
* @throws TableNotFoundException
|
||||
* @throws AccumuloSecurityException
|
||||
*/
|
||||
public void shutdown(boolean compact) throws AccumuloSecurityException, TableNotFoundException, AccumuloException {
|
||||
public void shutdown(boolean compact) throws AccumuloSecurityException,
|
||||
TableNotFoundException, AccumuloException {
|
||||
// Make sure this wasn't closed already.
|
||||
if (mtbw == null) {
|
||||
throw new RuntimeException("Ingester was already closed");
|
||||
}
|
||||
|
||||
mtbw.close();
|
||||
mtbw = null;
|
||||
|
||||
@@ -266,14 +269,14 @@ public final class AccumuloBulkIngester {
|
||||
* builder.finish();
|
||||
* </PRE>
|
||||
*/
|
||||
public final class PropertyBuilder {
|
||||
public static final class PropertyBuilder {
|
||||
|
||||
Mutation mutation;
|
||||
BatchWriter writer;
|
||||
final String id;
|
||||
final BatchWriter writer;
|
||||
|
||||
PropertyBuilder(BatchWriter writer, String id) {
|
||||
this.writer = writer;
|
||||
this.mutation = new Mutation(id);
|
||||
this.id = id;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -285,7 +288,13 @@ public final class AccumuloBulkIngester {
|
||||
* @return
|
||||
*/
|
||||
public PropertyBuilder add(String key, Object value) {
|
||||
mutation.put(key.getBytes(), AccumuloGraph.EMPTY, AccumuloByteSerializer.serialize(value));
|
||||
for (Mutation m : new WritePropertyMutator(id, key, value).create()) {
|
||||
try {
|
||||
writer.addMutation(m);
|
||||
} catch (MutationsRejectedException e) {
|
||||
throw new AccumuloGraphException(e);
|
||||
}
|
||||
}
|
||||
return this;
|
||||
}
|
||||
|
||||
@@ -295,9 +304,7 @@ public final class AccumuloBulkIngester {
|
||||
* @throws MutationsRejectedException
|
||||
*/
|
||||
public void finish() throws MutationsRejectedException {
|
||||
if (mutation.size() > 0) {
|
||||
writer.addMutation(mutation);
|
||||
}
|
||||
// No-op since Mutations are now added on the fly.
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -306,7 +313,7 @@ public final class AccumuloBulkIngester {
|
||||
* @return
|
||||
*/
|
||||
public String getId() {
|
||||
return new String(mutation.getRow());
|
||||
return id;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -26,21 +26,21 @@ import javax.xml.namespace.QName;
|
||||
|
||||
public final class AccumuloByteSerializer {
|
||||
|
||||
static final int NULL = 'n';
|
||||
public static final int NULL = 'n';
|
||||
|
||||
static final int BYTE = 'b';
|
||||
static final int SHORT = 's';
|
||||
static final int CHARACTER = 'c';
|
||||
static final int INTEGER = 'i';
|
||||
static final int LONG = 'l';
|
||||
static final int FLOAT = 'f';
|
||||
static final int DOUBLE = 'd';
|
||||
static final int BOOLEAN = 'o';
|
||||
static final int DATE = 't';
|
||||
static final int ENUM = 'e';
|
||||
static final int STRING = 'a';
|
||||
static final int SERIALIZABLE = 'x';
|
||||
static final int QNAME = 'q';
|
||||
public static final int BYTE = 'b';
|
||||
public static final int SHORT = 's';
|
||||
public static final int CHARACTER = 'c';
|
||||
public static final int INTEGER = 'i';
|
||||
public static final int LONG = 'l';
|
||||
public static final int FLOAT = 'f';
|
||||
public static final int DOUBLE = 'd';
|
||||
public static final int BOOLEAN = 'o';
|
||||
public static final int DATE = 't';
|
||||
public static final int ENUM = 'e';
|
||||
public static final int STRING = 'a';
|
||||
public static final int SERIALIZABLE = 'x';
|
||||
public static final int QNAME = 'q';
|
||||
|
||||
private AccumuloByteSerializer() {
|
||||
|
||||
@@ -53,7 +53,8 @@ public final class AccumuloByteSerializer {
|
||||
}
|
||||
};
|
||||
|
||||
public static <T> T desserialize(byte[] target) {
|
||||
@SuppressWarnings("unchecked")
|
||||
public static <T> T deserialize(byte[] target) {
|
||||
if (target[0] == NULL) {
|
||||
return null;
|
||||
}
|
||||
@@ -92,6 +93,7 @@ public final class AccumuloByteSerializer {
|
||||
case ENUM:
|
||||
try {
|
||||
String[] s = new String(target, 1, target.length - 1).split(":");
|
||||
@SuppressWarnings("rawtypes")
|
||||
Class<? extends Enum> clz = (Class<? extends Enum>) Class.forName(s[0]);
|
||||
return (T) Enum.valueOf(clz, s[1]);
|
||||
} catch (ClassNotFoundException cnfe) {
|
||||
@@ -107,6 +109,8 @@ public final class AccumuloByteSerializer {
|
||||
} catch (ClassNotFoundException cnfe) {
|
||||
throw new RuntimeException("Unexpected error deserializing object.", cnfe);
|
||||
}
|
||||
case NULL:
|
||||
return null;
|
||||
default:
|
||||
throw new RuntimeException("Unexpected data type: " + (char) target[0]);
|
||||
}
|
||||
|
||||
@@ -14,71 +14,55 @@
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop;
|
||||
|
||||
import java.util.Map;
|
||||
import java.util.Map.Entry;
|
||||
|
||||
import org.apache.log4j.Logger;
|
||||
|
||||
import com.tinkerpop.blueprints.Direction;
|
||||
import com.tinkerpop.blueprints.Edge;
|
||||
import com.tinkerpop.blueprints.Vertex;
|
||||
import com.tinkerpop.blueprints.util.ExceptionFactory;
|
||||
import com.tinkerpop.blueprints.util.StringFactory;
|
||||
|
||||
/**
|
||||
* TODO
|
||||
*/
|
||||
public class AccumuloEdge extends AccumuloElement implements Edge {
|
||||
|
||||
String label;
|
||||
String inId;
|
||||
String outId;
|
||||
Vertex inVertex;
|
||||
Vertex outVertex;
|
||||
private static final Logger log = Logger.getLogger(AccumuloEdge.class);
|
||||
|
||||
AccumuloEdge(AccumuloGraph parent, String id) {
|
||||
this(parent, id, null);
|
||||
private String label;
|
||||
private Vertex inVertex;
|
||||
private Vertex outVertex;
|
||||
|
||||
public AccumuloEdge(GlobalInstances globals, String id) {
|
||||
this(globals, id, null, null, null);
|
||||
}
|
||||
|
||||
AccumuloEdge(AccumuloGraph parent, String id, String label) {
|
||||
this(parent, id, label, (Vertex) null, (Vertex) null);
|
||||
}
|
||||
|
||||
AccumuloEdge(AccumuloGraph parent, String id, String label, Vertex inVertex, Vertex outVertex) {
|
||||
super(parent, id, Edge.class);
|
||||
public AccumuloEdge(GlobalInstances globals, String id,
|
||||
Vertex inVertex, Vertex outVertex, String label) {
|
||||
super(globals, id, Edge.class);
|
||||
this.label = label;
|
||||
this.inVertex = inVertex;
|
||||
this.outVertex = outVertex;
|
||||
}
|
||||
|
||||
AccumuloEdge(AccumuloGraph parent, String id, String label, String inVertex, String outVertex) {
|
||||
super(parent, id, Edge.class);
|
||||
this.label = label;
|
||||
this.inId = inVertex;
|
||||
this.outId = outVertex;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Vertex getVertex(Direction direction) throws IllegalArgumentException {
|
||||
switch (direction) {
|
||||
case IN:
|
||||
if (inVertex == null) {
|
||||
if (inId == null) {
|
||||
inVertex = parent.getEdgeVertex(id, direction);
|
||||
inId = inVertex.getId().toString();
|
||||
} else {
|
||||
inVertex = parent.getVertex(inId);
|
||||
}
|
||||
}
|
||||
return inVertex;
|
||||
case OUT:
|
||||
if (outVertex == null) {
|
||||
if (outId == null) {
|
||||
outVertex = parent.getEdgeVertex(id, direction);
|
||||
outId = outVertex.getId().toString();
|
||||
} else {
|
||||
outVertex = parent.getVertex(outId);
|
||||
}
|
||||
}
|
||||
return outVertex;
|
||||
case BOTH:
|
||||
throw ExceptionFactory.bothIsNotSupported();
|
||||
default:
|
||||
throw new RuntimeException("Unexpected direction: " + direction);
|
||||
if (!Direction.IN.equals(direction) && !Direction.OUT.equals(direction)) {
|
||||
throw new IllegalArgumentException("Invalid direction: "+direction);
|
||||
}
|
||||
|
||||
// The vertex information needs to be loaded.
|
||||
if (inVertex == null || outVertex == null || label == null) {
|
||||
log.debug("Loading information for edge: "+this);
|
||||
globals.getEdgeWrapper().loadEndpointsAndLabel(this);
|
||||
}
|
||||
|
||||
return Direction.IN.equals(direction) ? inVertex : outVertex;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String getLabel() {
|
||||
// TODO less special treatment for "LABEL" property...
|
||||
if (label != null) {
|
||||
@@ -87,31 +71,47 @@ public class AccumuloEdge extends AccumuloElement implements Edge {
|
||||
return getProperty(StringFactory.LABEL);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void remove() {
|
||||
parent.removeEdge(this);
|
||||
// Remove from named indexes.
|
||||
super.removeElementFromNamedIndexes();
|
||||
|
||||
// If edge was removed already, forget it.
|
||||
// This may happen due to self-loops...
|
||||
if (!globals.getEdgeWrapper().elementExists(id)) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Remove properties from key/value indexes.
|
||||
Map<String, Object> props = globals.getEdgeWrapper()
|
||||
.readAllProperties(this);
|
||||
|
||||
for (Entry<String,Object> ents : props.entrySet()) {
|
||||
globals.getEdgeKeyIndexWrapper().removePropertyFromIndex(this,
|
||||
ents.getKey(), ents.getValue());
|
||||
}
|
||||
|
||||
// Get rid of the endpoints and edge themselves.
|
||||
globals.getVertexWrapper().deleteEdgeEndpoints(this);
|
||||
globals.getEdgeWrapper().deleteEdge(this);
|
||||
|
||||
// Remove element from cache.
|
||||
globals.getCaches().remove(id, Edge.class);
|
||||
|
||||
globals.checkedFlush();
|
||||
}
|
||||
|
||||
public String getInId() {
|
||||
return inId;
|
||||
public void setVertices(AccumuloVertex inVertex, AccumuloVertex outVertex) {
|
||||
this.inVertex = inVertex;
|
||||
this.outVertex = outVertex;
|
||||
}
|
||||
|
||||
public String getOutId() {
|
||||
return outId;
|
||||
}
|
||||
|
||||
protected void setInId(String id){
|
||||
inId = id;
|
||||
}
|
||||
|
||||
protected void setOutId(String id){
|
||||
outId = id;
|
||||
}
|
||||
|
||||
protected void setLabel(String label){
|
||||
public void setLabel(String label) {
|
||||
this.label = label;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString() {
|
||||
return "[" + getId() + ":" + getVertex(Direction.OUT) + " -> " + getLabel() + " -> " + getVertex(Direction.IN) + "]";
|
||||
return "[" + getId() + ":" + inVertex + " -> " + label + " -> " + outVertex + "]";
|
||||
}
|
||||
}
|
||||
|
||||
@@ -14,84 +14,163 @@
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop;
|
||||
|
||||
import java.util.HashMap;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
|
||||
import org.apache.accumulo.core.util.Pair;
|
||||
|
||||
import com.tinkerpop.blueprints.Element;
|
||||
import com.tinkerpop.blueprints.Index;
|
||||
import com.tinkerpop.blueprints.util.StringFactory;
|
||||
|
||||
import edu.jhuapl.tinkerpop.cache.PropertyCache;
|
||||
|
||||
/**
|
||||
* TODO
|
||||
*/
|
||||
public abstract class AccumuloElement implements Element {
|
||||
|
||||
protected AccumuloGraph parent;
|
||||
protected GlobalInstances globals;
|
||||
protected String id;
|
||||
|
||||
private Class type;
|
||||
private Class<? extends Element> type;
|
||||
|
||||
private Map<String,Pair<Long,Object>> propertiesCache;
|
||||
private PropertyCache propertyCache;
|
||||
|
||||
protected AccumuloElement(AccumuloGraph parent, String id, Class<? extends Element> type) {
|
||||
this.parent = parent;
|
||||
protected AccumuloElement(GlobalInstances globals,
|
||||
String id, Class<? extends Element> type) {
|
||||
this.globals = globals;
|
||||
this.id = id;
|
||||
this.type = type;
|
||||
}
|
||||
|
||||
public <T> T getProperty(String key) {
|
||||
if (propertiesCache == null) {
|
||||
// lazily create the properties cache...
|
||||
/**
|
||||
* Create properties cache if it doesn't exist,
|
||||
* and preload any properties.
|
||||
*/
|
||||
private void makeCache() {
|
||||
if (propertyCache == null) {
|
||||
propertyCache = new PropertyCache(globals.getConfig());
|
||||
|
||||
// we will create it here just in case the parent does not actually
|
||||
// pre-load any data. Note it also may be created in the
|
||||
// cacheProperty method, as well, in the event a class pre-loads
|
||||
// data before a call is made to obtain it.
|
||||
propertiesCache = new HashMap<String,Pair<Long,Object>>();
|
||||
|
||||
parent.preloadProperties(this, type);
|
||||
}
|
||||
|
||||
Pair<Long,Object> val = propertiesCache.get(key);
|
||||
if (val != null) {
|
||||
if (val.getFirst() < System.currentTimeMillis()) {
|
||||
// this cached value has timed out..
|
||||
propertiesCache.remove(key);
|
||||
} else {
|
||||
// the cached value is still good...
|
||||
return (T) val.getSecond();
|
||||
// Preload any keys, if needed.
|
||||
String[] preloadKeys = globals.getConfig().getPreloadedProperties();
|
||||
if (preloadKeys != null) {
|
||||
propertyCache.putAll(globals.getElementWrapper(type)
|
||||
.readProperties(this, preloadKeys));
|
||||
}
|
||||
}
|
||||
Pair<Integer,T> pair = parent.getProperty(type, id, key);
|
||||
if (pair.getFirst() != null) {
|
||||
cacheProperty(key, pair.getSecond(), pair.getFirst());
|
||||
}
|
||||
return pair.getSecond();
|
||||
}
|
||||
|
||||
@Override
|
||||
public <T> T getProperty(String key) {
|
||||
makeCache();
|
||||
|
||||
// Get from property cache.
|
||||
T value = propertyCache.get(key);
|
||||
|
||||
// If not cached, get it from the backing table.
|
||||
if (value == null) {
|
||||
value = globals.getElementWrapper(type).readProperty(this, key);
|
||||
}
|
||||
|
||||
// Cache the new value.
|
||||
if (value != null) {
|
||||
propertyCache.put(key, value);
|
||||
}
|
||||
|
||||
return value;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Set<String> getPropertyKeys() {
|
||||
return parent.getPropertyKeys(type, id);
|
||||
return globals.getElementWrapper(type).readPropertyKeys(this);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void setProperty(String key, Object value) {
|
||||
Integer timeout = parent.setProperty(type, id, key, value);
|
||||
cacheProperty(key, value, timeout);
|
||||
makeCache();
|
||||
globals.getKeyIndexTableWrapper(type).setPropertyForIndex(this, key, value);
|
||||
// MDL 31 Dec 2014: The above calls getProperty, so this
|
||||
// order is important (for now).
|
||||
globals.getElementWrapper(type).writeProperty(this, key, value);
|
||||
globals.checkedFlush();
|
||||
setPropertyInMemory(key, value);
|
||||
}
|
||||
|
||||
/**
|
||||
* Set a property but only in the instantiated object,
|
||||
* not in the backing store.
|
||||
* @param key
|
||||
* @param value
|
||||
*/
|
||||
public void setPropertyInMemory(String key, Object value) {
|
||||
makeCache();
|
||||
propertyCache.put(key, value);
|
||||
}
|
||||
|
||||
@Override
|
||||
public <T> T removeProperty(String key) {
|
||||
if (propertiesCache != null) {
|
||||
// we have the cached value but we still need to pass this on to the
|
||||
// parent so it can actually remove the data from the backing store.
|
||||
// Since we have to do that anyway, we will use the parent's value
|
||||
// instead of the cache value to to be as up-to-date as possible.
|
||||
// Of course we still need to clear out the cached value...
|
||||
propertiesCache.remove(key);
|
||||
if (StringFactory.LABEL.equals(key) ||
|
||||
Constants.LABEL.equals(key)) {
|
||||
throw new AccumuloGraphException("Cannot remove the " + StringFactory.LABEL + " property.");
|
||||
}
|
||||
return parent.removeProperty(type, id, key);
|
||||
|
||||
makeCache();
|
||||
T value = getProperty(key);
|
||||
if (value != null) {
|
||||
globals.getElementWrapper(type).clearProperty(this, key);
|
||||
globals.checkedFlush();
|
||||
}
|
||||
globals.getKeyIndexTableWrapper(type).removePropertyFromIndex(this, key, value);
|
||||
// MDL 31 Dec 2014: AccumuloGraph.removeProperty
|
||||
// calls getProperty which populates the cache.
|
||||
// So the order here is important (for now).
|
||||
removePropertyInMemory(key);
|
||||
return value;
|
||||
}
|
||||
|
||||
/**
|
||||
* Remove element from all named indexes.
|
||||
* @param element
|
||||
*/
|
||||
protected void removeElementFromNamedIndexes() {
|
||||
for (Index<? extends Element> index : globals.getIndexMetadataWrapper().getIndices()) {
|
||||
((AccumuloIndex<? extends Element>) index).getWrapper().removeElementFromIndex(this);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Remove a property but only in the instantiated
|
||||
* object, not the backing store.
|
||||
* @param key
|
||||
*/
|
||||
public void removePropertyInMemory(String key) {
|
||||
makeCache();
|
||||
propertyCache.remove(key);
|
||||
}
|
||||
|
||||
/**
|
||||
* Return the properties currently cached in memory.
|
||||
* @return
|
||||
*/
|
||||
public Iterable<String> getPropertyKeysInMemory() {
|
||||
makeCache();
|
||||
return propertyCache.keySet();
|
||||
}
|
||||
|
||||
/**
|
||||
* Retrieve a property from memory.
|
||||
* @param key
|
||||
* @return
|
||||
*/
|
||||
public Object getPropertyInMemory(String key) {
|
||||
makeCache();
|
||||
return propertyCache.get(key);
|
||||
}
|
||||
|
||||
@Override
|
||||
public Object getId() {
|
||||
return id;
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean equals(Object obj) {
|
||||
if (obj == null) {
|
||||
return false;
|
||||
@@ -100,25 +179,20 @@ public abstract class AccumuloElement implements Element {
|
||||
} else if (!obj.getClass().equals(getClass())) {
|
||||
return false;
|
||||
} else {
|
||||
return this.id.equals(((AccumuloElement) obj).id);
|
||||
return id.equals(((AccumuloElement) obj).id);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public int hashCode() {
|
||||
return getClass().hashCode() ^ id.hashCode();
|
||||
}
|
||||
|
||||
void cacheProperty(String key, Object value, Integer timeoutMillis) {
|
||||
if (timeoutMillis == null) {
|
||||
// user does not want to cache data...
|
||||
return;
|
||||
}
|
||||
|
||||
if (propertiesCache == null) {
|
||||
propertiesCache = new HashMap<String,Pair<Long,Object>>();
|
||||
}
|
||||
Pair<Long,Object> tsVal = new Pair<Long,Object>(System.currentTimeMillis() + timeoutMillis, value);
|
||||
propertiesCache.put(key, tsVal);
|
||||
/**
|
||||
* Internal method for unit tests.
|
||||
* @return
|
||||
*/
|
||||
PropertyCache getPropertyCache() {
|
||||
return propertyCache;
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
62
src/main/java/edu/jhuapl/tinkerpop/AccumuloFeatures.java
Normal file
62
src/main/java/edu/jhuapl/tinkerpop/AccumuloFeatures.java
Normal file
@@ -0,0 +1,62 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop;
|
||||
|
||||
import com.tinkerpop.blueprints.Features;
|
||||
|
||||
/**
|
||||
* {@link Features} creator.
|
||||
*/
|
||||
public class AccumuloFeatures {
|
||||
|
||||
public static Features get() {
|
||||
Features f = new Features();
|
||||
|
||||
// For simplicity, I accept all property types. They are handled in not the
|
||||
// best way. To be fixed later.
|
||||
f.ignoresSuppliedIds = true;
|
||||
f.isPersistent = true;
|
||||
f.isWrapper = false;
|
||||
f.supportsBooleanProperty = true;
|
||||
f.supportsDoubleProperty = true;
|
||||
f.supportsDuplicateEdges = true;
|
||||
f.supportsEdgeIndex = true;
|
||||
f.supportsEdgeIteration = true;
|
||||
f.supportsEdgeRetrieval = true;
|
||||
f.supportsEdgeKeyIndex = true;
|
||||
f.supportsEdgeProperties = true;
|
||||
f.supportsFloatProperty = true;
|
||||
f.supportsIndices = true;
|
||||
f.supportsIntegerProperty = true;
|
||||
f.supportsKeyIndices = true;
|
||||
f.supportsLongProperty = true;
|
||||
f.supportsMapProperty = true;
|
||||
f.supportsMixedListProperty = true;
|
||||
f.supportsPrimitiveArrayProperty = true;
|
||||
f.supportsSelfLoops = true;
|
||||
f.supportsSerializableObjectProperty = true;
|
||||
f.supportsStringProperty = true;
|
||||
f.supportsThreadedTransactions = false;
|
||||
f.supportsTransactions = false;
|
||||
f.supportsUniformListProperty = true;
|
||||
f.supportsVertexIndex = true;
|
||||
f.supportsVertexIteration = true;
|
||||
f.supportsVertexKeyIndex = true;
|
||||
f.supportsVertexProperties = true;
|
||||
f.supportsThreadIsolatedTransactions = false;
|
||||
|
||||
return f;
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -14,36 +14,16 @@
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.SortedSet;
|
||||
import java.util.UUID;
|
||||
|
||||
import org.apache.accumulo.core.client.AccumuloException;
|
||||
import org.apache.accumulo.core.client.AccumuloSecurityException;
|
||||
import org.apache.accumulo.core.client.TableExistsException;
|
||||
import org.apache.accumulo.core.client.TableNotFoundException;
|
||||
import org.apache.accumulo.core.client.admin.TableOperations;
|
||||
import org.apache.accumulo.core.data.Value;
|
||||
import org.apache.hadoop.io.Text;
|
||||
|
||||
import com.tinkerpop.blueprints.util.ExceptionFactory;
|
||||
import com.tinkerpop.blueprints.util.StringFactory;
|
||||
|
||||
final class AccumuloGraphUtils {
|
||||
|
||||
public static final Value EMPTY_VALUE = new Value(new byte[0]);
|
||||
public static final Text EMPTY_TEXT = new Text("");
|
||||
|
||||
public static final Text ID = new Text(StringFactory.ID);
|
||||
|
||||
public static final Text IN = new Text("I");
|
||||
public static final Text OUT = new Text("O");
|
||||
|
||||
public static final String toId(Object obj) {
|
||||
return obj.toString();
|
||||
}
|
||||
|
||||
public static Value toValue(String val) {
|
||||
return new Value(val.getBytes());
|
||||
}
|
||||
public final class AccumuloGraphUtils {
|
||||
|
||||
/**
|
||||
* Create and/or clear existing graph tables for the given configuration.
|
||||
@@ -65,17 +45,17 @@ final class AccumuloGraphUtils {
|
||||
|
||||
// Check edge cases.
|
||||
// No tables exist, and we are not allowed to create.
|
||||
if (!existedBeforeClear && !cfg.isCreate()) {
|
||||
if (!existedBeforeClear && !cfg.getCreate()) {
|
||||
throw new IllegalArgumentException("Graph does not exist, and create option is disabled");
|
||||
}
|
||||
// Tables exist, and we are not clearing them.
|
||||
else if (existedBeforeClear && !cfg.isClear()) {
|
||||
else if (existedBeforeClear && !cfg.getClear()) {
|
||||
// Do nothing.
|
||||
return;
|
||||
}
|
||||
|
||||
// We want to clear tables, so do it.
|
||||
if (cfg.isClear()) {
|
||||
if (cfg.getClear()) {
|
||||
for (String table : cfg.getTableNames()) {
|
||||
if (tableOps.exists(table)) {
|
||||
tableOps.delete(table);
|
||||
@@ -84,7 +64,7 @@ final class AccumuloGraphUtils {
|
||||
}
|
||||
|
||||
// Tables existed, or we want to create them. So do it.
|
||||
if (existedBeforeClear || cfg.isCreate()) {
|
||||
if (existedBeforeClear || cfg.getCreate()) {
|
||||
for (String table : cfg.getTableNames()) {
|
||||
if (!tableOps.exists(table)) {
|
||||
tableOps.create(table);
|
||||
@@ -96,18 +76,49 @@ final class AccumuloGraphUtils {
|
||||
}
|
||||
}
|
||||
|
||||
} catch (AccumuloException e) {
|
||||
throw new IllegalArgumentException(e);
|
||||
} catch (AccumuloSecurityException e) {
|
||||
throw new IllegalArgumentException(e);
|
||||
} catch (IOException e) {
|
||||
throw new IllegalArgumentException(e);
|
||||
} catch (InterruptedException e) {
|
||||
throw new IllegalArgumentException(e);
|
||||
} catch (TableNotFoundException e) {
|
||||
throw new IllegalArgumentException(e);
|
||||
} catch (TableExistsException e) {
|
||||
} catch (Exception e) {
|
||||
throw new IllegalArgumentException(e);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Generate an element id.
|
||||
* @return
|
||||
*/
|
||||
public static String generateId() {
|
||||
return UUID.randomUUID().toString();
|
||||
}
|
||||
|
||||
/**
|
||||
* Ensure that the given key/value don't conflict with
|
||||
* Blueprints reserved words.
|
||||
* @param key
|
||||
* @param value
|
||||
*/
|
||||
public static void validateProperty(String key, Object value) {
|
||||
nullCheckProperty(key, value);
|
||||
if (key.equals(StringFactory.ID)) {
|
||||
throw ExceptionFactory.propertyKeyIdIsReserved();
|
||||
} else if (key.equals(StringFactory.LABEL)) {
|
||||
throw ExceptionFactory.propertyKeyLabelIsReservedForEdges();
|
||||
} else if (value == null) {
|
||||
throw ExceptionFactory.propertyValueCanNotBeNull();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Disallow null keys/values and throw appropriate
|
||||
* Blueprints exceptions.
|
||||
* @param key
|
||||
* @param value
|
||||
*/
|
||||
public static void nullCheckProperty(String key, Object value) {
|
||||
if (key == null) {
|
||||
throw ExceptionFactory.propertyKeyCanNotBeNull();
|
||||
} else if (value == null) {
|
||||
throw ExceptionFactory.propertyValueCanNotBeNull();
|
||||
} else if (key.trim().equals(StringFactory.EMPTY_STRING)) {
|
||||
throw ExceptionFactory.propertyKeyCanNotBeEmpty();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -14,87 +14,77 @@
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.Iterator;
|
||||
import java.util.Map.Entry;
|
||||
|
||||
import org.apache.accumulo.core.client.AccumuloException;
|
||||
import org.apache.accumulo.core.client.AccumuloSecurityException;
|
||||
import org.apache.accumulo.core.client.BatchWriter;
|
||||
import org.apache.accumulo.core.client.MutationsRejectedException;
|
||||
import org.apache.accumulo.core.client.Scanner;
|
||||
import org.apache.accumulo.core.client.ScannerBase;
|
||||
import org.apache.accumulo.core.client.TableExistsException;
|
||||
import org.apache.accumulo.core.data.Key;
|
||||
import org.apache.accumulo.core.data.Mutation;
|
||||
import org.apache.accumulo.core.data.Range;
|
||||
import org.apache.accumulo.core.data.Value;
|
||||
import org.apache.accumulo.core.util.PeekingIterator;
|
||||
import org.apache.hadoop.io.Text;
|
||||
|
||||
import com.tinkerpop.blueprints.CloseableIterable;
|
||||
import com.tinkerpop.blueprints.Edge;
|
||||
import com.tinkerpop.blueprints.Element;
|
||||
import com.tinkerpop.blueprints.Index;
|
||||
import com.tinkerpop.blueprints.IndexableGraph;
|
||||
|
||||
import edu.jhuapl.tinkerpop.tables.index.NamedIndexTableWrapper;
|
||||
|
||||
/**
|
||||
* Accumulo-based implementation for {@link IndexableGraph}.
|
||||
* @param <T>
|
||||
*/
|
||||
public class AccumuloIndex<T extends Element> implements Index<T> {
|
||||
Class indexedType;
|
||||
AccumuloGraph parent;
|
||||
String indexName;
|
||||
String tableName;
|
||||
private final GlobalInstances globals;
|
||||
private final Class<T> indexedType;
|
||||
private final String indexName;
|
||||
private final NamedIndexTableWrapper indexWrapper;
|
||||
|
||||
public AccumuloIndex(Class t, AccumuloGraph parent, String indexName) {
|
||||
indexedType = t;
|
||||
this.parent = parent;
|
||||
public AccumuloIndex(GlobalInstances globals, String indexName, Class<T> indexedType) {
|
||||
this.globals = globals;
|
||||
this.indexName = indexName;
|
||||
tableName = parent.config.getName() + "_index_" + indexName;// + "_" +
|
||||
// t;
|
||||
this.indexedType = indexedType;
|
||||
|
||||
try {
|
||||
if (!parent.config.getConnector().tableOperations().exists(tableName)) {
|
||||
parent.config.getConnector().tableOperations().create(tableName);
|
||||
if (!globals.getConfig().getConnector()
|
||||
.tableOperations().exists(getTableName())) {
|
||||
globals.getConfig().getConnector()
|
||||
.tableOperations().create(getTableName());
|
||||
}
|
||||
} catch (Exception e) {
|
||||
throw new RuntimeException(e);
|
||||
throw new AccumuloGraphException(e);
|
||||
}
|
||||
|
||||
indexWrapper = new NamedIndexTableWrapper(globals, indexedType, indexName);
|
||||
}
|
||||
|
||||
@Override
|
||||
public String getIndexName() {
|
||||
return indexName;
|
||||
}
|
||||
|
||||
public String getTableName() {
|
||||
return globals.getConfig().getNamedIndexTableName(indexName);
|
||||
}
|
||||
|
||||
public NamedIndexTableWrapper getWrapper() {
|
||||
return indexWrapper;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Class<T> getIndexClass() {
|
||||
return indexedType;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void put(String key, Object value, Element element) {
|
||||
element.setProperty(key, value);
|
||||
Mutation m = new Mutation(AccumuloByteSerializer.serialize(value));
|
||||
m.put(key.getBytes(), element.getId().toString().getBytes(), "".getBytes());
|
||||
BatchWriter w = getWriter();
|
||||
try {
|
||||
w.addMutation(m);
|
||||
w.flush();
|
||||
} catch (MutationsRejectedException e) {
|
||||
e.printStackTrace();
|
||||
}
|
||||
|
||||
indexWrapper.setPropertyForIndex(element, key, value, true);
|
||||
}
|
||||
|
||||
@Override
|
||||
public CloseableIterable<T> get(String key, Object value) {
|
||||
Scanner scan = getScanner();
|
||||
byte[] id = AccumuloByteSerializer.serialize(value);
|
||||
scan.setRange(new Range(new Text(id), new Text(id)));
|
||||
scan.fetchColumnFamily(new Text(key));
|
||||
|
||||
return new IndexIterable(parent, scan, indexedType);
|
||||
return indexWrapper.readElementsFromIndex(key, value);
|
||||
}
|
||||
|
||||
@Override
|
||||
public CloseableIterable<T> query(String key, Object query) {
|
||||
throw new UnsupportedOperationException();
|
||||
|
||||
}
|
||||
|
||||
@Override
|
||||
public long count(String key, Object value) {
|
||||
CloseableIterable<T> iterable = get(key, value);
|
||||
Iterator<T> iter = iterable.iterator();
|
||||
@@ -107,80 +97,8 @@ public class AccumuloIndex<T extends Element> implements Index<T> {
|
||||
return count;
|
||||
}
|
||||
|
||||
public void remove(String key, Object value, Element element) {
|
||||
Mutation m = new Mutation(AccumuloByteSerializer.serialize(value));
|
||||
m.putDelete(key.getBytes(), element.getId().toString().getBytes());
|
||||
BatchWriter w = getWriter();
|
||||
try {
|
||||
w.addMutation(m);
|
||||
w.flush();
|
||||
} catch (MutationsRejectedException e) {
|
||||
e.printStackTrace();
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
private BatchWriter getWriter() {
|
||||
return parent.getWriter(tableName);
|
||||
}
|
||||
|
||||
private Scanner getScanner() {
|
||||
return parent.getScanner(tableName);
|
||||
}
|
||||
|
||||
public class IndexIterable implements CloseableIterable<T> {
|
||||
AccumuloGraph parent;
|
||||
ScannerBase scan;
|
||||
boolean isClosed;
|
||||
Class indexedType;
|
||||
|
||||
IndexIterable(AccumuloGraph parent, ScannerBase scan, Class t) {
|
||||
this.scan = scan;
|
||||
this.parent = parent;
|
||||
isClosed = false;
|
||||
indexedType = t;
|
||||
}
|
||||
|
||||
public Iterator<T> iterator() {
|
||||
if (!isClosed) {
|
||||
if(indexedType.equals(Edge.class)){
|
||||
|
||||
return new ScannerIterable<T>(parent, scan) {
|
||||
|
||||
@Override
|
||||
public T next(PeekingIterator<Entry<Key,Value>> iterator) {
|
||||
// TODO better use of information readily
|
||||
// available...
|
||||
return (T) new AccumuloEdge(parent, iterator.next().getKey().getColumnQualifier().toString());
|
||||
}
|
||||
}.iterator();
|
||||
}else{
|
||||
return new ScannerIterable<T>(parent, scan) {
|
||||
|
||||
@Override
|
||||
public T next(PeekingIterator<Entry<Key,Value>> iterator) {
|
||||
// TODO better use of information readily
|
||||
// available...
|
||||
return (T) new AccumuloVertex(parent, iterator.next().getKey().getColumnQualifier().toString());
|
||||
}
|
||||
}.iterator();
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
public void close() {
|
||||
if (!isClosed) {
|
||||
scan.close();
|
||||
isClosed = true;
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@Override
|
||||
public Class<T> getIndexClass() {
|
||||
return indexedType;
|
||||
public void remove(String key, Object value, Element element) {
|
||||
indexWrapper.removePropertyFromIndex(element, key, value);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@@ -14,38 +14,114 @@
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop;
|
||||
|
||||
import java.util.Map;
|
||||
import java.util.Map.Entry;
|
||||
|
||||
import com.tinkerpop.blueprints.CloseableIterable;
|
||||
import com.tinkerpop.blueprints.Direction;
|
||||
import com.tinkerpop.blueprints.Edge;
|
||||
import com.tinkerpop.blueprints.Vertex;
|
||||
import com.tinkerpop.blueprints.VertexQuery;
|
||||
import com.tinkerpop.blueprints.util.DefaultVertexQuery;
|
||||
import com.tinkerpop.blueprints.util.ExceptionFactory;
|
||||
|
||||
/**
|
||||
* TODO
|
||||
*/
|
||||
public class AccumuloVertex extends AccumuloElement implements Vertex {
|
||||
|
||||
AccumuloVertex(AccumuloGraph parent, String id) {
|
||||
super(parent, id, Vertex.class);
|
||||
public AccumuloVertex(GlobalInstances globals, String id) {
|
||||
super(globals, id, Vertex.class);
|
||||
}
|
||||
|
||||
@Override
|
||||
public Iterable<Edge> getEdges(Direction direction, String... labels) {
|
||||
return parent.getEdges(id, direction, labels);
|
||||
return globals.getVertexWrapper().getEdges(this, direction, labels);
|
||||
}
|
||||
|
||||
@Override
|
||||
public Iterable<Vertex> getVertices(Direction direction, String... labels) {
|
||||
return parent.getVertices(id, direction, labels);
|
||||
return globals.getVertexWrapper().getVertices(this, direction, labels);
|
||||
}
|
||||
|
||||
@Override
|
||||
public VertexQuery query() {
|
||||
return new DefaultVertexQuery(this);
|
||||
}
|
||||
|
||||
@Override
|
||||
public Edge addEdge(String label, Vertex inVertex) {
|
||||
return parent.addEdge(null, this, inVertex, label);
|
||||
return addEdge(null, label, inVertex);
|
||||
}
|
||||
|
||||
/**
|
||||
* Add an edge as with {@link #addEdge(String, Vertex)},
|
||||
* but with a specified edge id.
|
||||
* @param id
|
||||
* @param label
|
||||
* @param inVertex
|
||||
* @return
|
||||
*/
|
||||
public Edge addEdge(Object id, String label, Vertex inVertex) {
|
||||
if (label == null) {
|
||||
throw ExceptionFactory.edgeLabelCanNotBeNull();
|
||||
}
|
||||
if (id == null) {
|
||||
id = AccumuloGraphUtils.generateId();
|
||||
}
|
||||
|
||||
String myID = id.toString();
|
||||
|
||||
AccumuloEdge edge = new AccumuloEdge(globals, myID, inVertex, this, label);
|
||||
|
||||
// TODO we arent suppose to make sure the given edge ID doesn't already
|
||||
// exist?
|
||||
|
||||
globals.getEdgeWrapper().writeEdge(edge);
|
||||
globals.getVertexWrapper().writeEdgeEndpoints(edge);
|
||||
|
||||
globals.checkedFlush();
|
||||
|
||||
globals.getCaches().cache(edge, Edge.class);
|
||||
|
||||
return edge;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void remove() {
|
||||
parent.removeVertex(this);
|
||||
globals.getCaches().remove(getId(), Vertex.class);
|
||||
|
||||
super.removeElementFromNamedIndexes();
|
||||
|
||||
// Throw exception if the element does not exist.
|
||||
if (!globals.getVertexWrapper().elementExists(id)) {
|
||||
throw ExceptionFactory.vertexWithIdDoesNotExist(getId());
|
||||
}
|
||||
|
||||
// Remove properties from key/value indexes.
|
||||
Map<String, Object> props = globals.getVertexWrapper()
|
||||
.readAllProperties(this);
|
||||
|
||||
for (Entry<String,Object> ent : props.entrySet()) {
|
||||
globals.getVertexKeyIndexWrapper().removePropertyFromIndex(this,
|
||||
ent.getKey(), ent.getValue());
|
||||
}
|
||||
|
||||
// Remove edges incident to this vertex.
|
||||
CloseableIterable<Edge> iter = (CloseableIterable<Edge>)getEdges(Direction.BOTH);
|
||||
for (Edge edge : iter) {
|
||||
edge.remove();
|
||||
}
|
||||
iter.close();
|
||||
|
||||
globals.checkedFlush();
|
||||
|
||||
// Get rid of the vertex.
|
||||
globals.getVertexWrapper().deleteVertex(this);
|
||||
globals.checkedFlush();
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString() {
|
||||
return "[" + getId() + "]";
|
||||
}
|
||||
|
||||
46
src/main/java/edu/jhuapl/tinkerpop/Constants.java
Normal file
46
src/main/java/edu/jhuapl/tinkerpop/Constants.java
Normal file
@@ -0,0 +1,46 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop;
|
||||
|
||||
/**
|
||||
* Collect up various constants here.
|
||||
* @author Michael Lieberman
|
||||
*
|
||||
*/
|
||||
public class Constants {
|
||||
|
||||
private Constants() { }
|
||||
|
||||
/**
|
||||
* Separate element ids in Accumulo entries.
|
||||
*/
|
||||
public static final String ID_DELIM = "__DELIM__";
|
||||
|
||||
public static final byte[] EMPTY = new byte[0];
|
||||
|
||||
/**
|
||||
* Prefixes for various Accumulo entries.
|
||||
*/
|
||||
public static final String LABEL = "__LABEL__";
|
||||
public static final String IN_EDGE = "__IN_EDGE__";
|
||||
public static final String OUT_EDGE = "__OUT_EDGE__";
|
||||
public static final String EXISTS = "__EXISTS__";
|
||||
|
||||
/**
|
||||
* Type of metadata to distinguish between
|
||||
* entries in the metadata table.
|
||||
*/
|
||||
public static enum IndexMetadataEntryType {__INDEX_KEY__, __INDEX_NAME__};
|
||||
}
|
||||
114
src/main/java/edu/jhuapl/tinkerpop/GlobalInstances.java
Normal file
114
src/main/java/edu/jhuapl/tinkerpop/GlobalInstances.java
Normal file
@@ -0,0 +1,114 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop;
|
||||
|
||||
import org.apache.accumulo.core.client.MultiTableBatchWriter;
|
||||
import org.apache.accumulo.core.client.MutationsRejectedException;
|
||||
|
||||
import com.tinkerpop.blueprints.Edge;
|
||||
import com.tinkerpop.blueprints.Element;
|
||||
import com.tinkerpop.blueprints.Vertex;
|
||||
|
||||
import edu.jhuapl.tinkerpop.cache.ElementCaches;
|
||||
import edu.jhuapl.tinkerpop.tables.core.EdgeTableWrapper;
|
||||
import edu.jhuapl.tinkerpop.tables.core.ElementTableWrapper;
|
||||
import edu.jhuapl.tinkerpop.tables.core.VertexTableWrapper;
|
||||
import edu.jhuapl.tinkerpop.tables.index.BaseKeyIndexTableWrapper;
|
||||
import edu.jhuapl.tinkerpop.tables.index.EdgeKeyIndexTableWrapper;
|
||||
import edu.jhuapl.tinkerpop.tables.index.IndexMetadataTableWrapper;
|
||||
import edu.jhuapl.tinkerpop.tables.index.VertexKeyIndexTableWrapper;
|
||||
|
||||
/**
|
||||
* Internal class gathering together instances of
|
||||
* objects needed for various AccumuloGraph components.
|
||||
*/
|
||||
public class GlobalInstances {
|
||||
|
||||
private final AccumuloGraphConfiguration config;
|
||||
private final MultiTableBatchWriter mtbw;
|
||||
private final ElementCaches caches;
|
||||
|
||||
public GlobalInstances(AccumuloGraphConfiguration config,
|
||||
MultiTableBatchWriter mtbw, ElementCaches caches) {
|
||||
this.config = config;
|
||||
this.mtbw = mtbw;
|
||||
this.caches = caches;
|
||||
}
|
||||
|
||||
public AccumuloGraphConfiguration getConfig() {
|
||||
return config;
|
||||
}
|
||||
|
||||
public MultiTableBatchWriter getMtbw() {
|
||||
return mtbw;
|
||||
}
|
||||
|
||||
public VertexTableWrapper getVertexWrapper() {
|
||||
return new VertexTableWrapper(this);
|
||||
}
|
||||
|
||||
public EdgeTableWrapper getEdgeWrapper() {
|
||||
return new EdgeTableWrapper(this);
|
||||
}
|
||||
|
||||
public VertexKeyIndexTableWrapper getVertexKeyIndexWrapper() {
|
||||
return new VertexKeyIndexTableWrapper(this);
|
||||
}
|
||||
|
||||
public EdgeKeyIndexTableWrapper getEdgeKeyIndexWrapper() {
|
||||
return new EdgeKeyIndexTableWrapper(this);
|
||||
}
|
||||
|
||||
public IndexMetadataTableWrapper getIndexMetadataWrapper() {
|
||||
return new IndexMetadataTableWrapper(this);
|
||||
}
|
||||
|
||||
public <T extends Element> ElementTableWrapper getElementWrapper(Class<T> clazz) {
|
||||
if (Vertex.class.equals(clazz)) {
|
||||
return getVertexWrapper();
|
||||
} else if (Edge.class.equals(clazz)) {
|
||||
return getEdgeWrapper();
|
||||
} else {
|
||||
throw new AccumuloGraphException("Unrecognized class: "+clazz);
|
||||
}
|
||||
}
|
||||
|
||||
public <T extends Element> BaseKeyIndexTableWrapper getKeyIndexTableWrapper(Class<T> clazz) {
|
||||
if (Vertex.class.equals(clazz)) {
|
||||
return getVertexKeyIndexWrapper();
|
||||
} else if (Edge.class.equals(clazz)) {
|
||||
return getEdgeKeyIndexWrapper();
|
||||
} else {
|
||||
throw new AccumuloGraphException("Unrecognized class: "+clazz);
|
||||
}
|
||||
}
|
||||
|
||||
public ElementCaches getCaches() {
|
||||
return caches;
|
||||
}
|
||||
|
||||
/**
|
||||
* Flush the writer, if autoflush is enabled.
|
||||
*/
|
||||
public void checkedFlush() {
|
||||
if (config.getAutoFlush()) {
|
||||
try {
|
||||
mtbw.flush();
|
||||
} catch (MutationsRejectedException e) {
|
||||
throw new AccumuloGraphException(e);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,122 +0,0 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop;
|
||||
|
||||
import java.lang.ref.ReferenceQueue;
|
||||
import java.lang.ref.SoftReference;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.Map;
|
||||
|
||||
import com.tinkerpop.blueprints.Element;
|
||||
|
||||
class LruElementCache<T extends Element> extends LinkedHashMap<String,LruElementCache<T>.Entry> implements Runnable {
|
||||
|
||||
private static final long serialVersionUID = 1435352624360026357L;
|
||||
|
||||
ReferenceQueue<T> queue;
|
||||
Integer maxCapacity;
|
||||
int timeout;
|
||||
|
||||
public LruElementCache(int timeout) {
|
||||
super(32, .75f, true);
|
||||
this.maxCapacity = null;
|
||||
init(timeout);
|
||||
}
|
||||
|
||||
public LruElementCache(int maxCapacity, int timeout) {
|
||||
super(maxCapacity + 1, 1f, true);
|
||||
this.maxCapacity = maxCapacity;
|
||||
init(timeout);
|
||||
}
|
||||
|
||||
private void init(int timeout) {
|
||||
this.timeout = timeout;
|
||||
this.queue = new ReferenceQueue<T>();
|
||||
|
||||
Thread t = new Thread(this, "lru-cache-reaper-" + System.identityHashCode(this));
|
||||
t.setPriority(Thread.MIN_PRIORITY);
|
||||
t.setDaemon(true);
|
||||
t.start();
|
||||
}
|
||||
|
||||
@Override
|
||||
protected boolean removeEldestEntry(Map.Entry<String,LruElementCache<T>.Entry> eldest) {
|
||||
if (maxCapacity != null) {
|
||||
return size() > maxCapacity;
|
||||
} else {
|
||||
// no cap on size, but see if eldest has timed out or been gc'ed
|
||||
// already...
|
||||
return eldest.getValue().getElement() == null;
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public synchronized Entry remove(Object id) {
|
||||
return super.remove(id);
|
||||
}
|
||||
|
||||
public synchronized void cache(T element) {
|
||||
Entry entry = new Entry(element, queue, timeout);
|
||||
put(element.getId().toString(), entry);
|
||||
}
|
||||
|
||||
public synchronized T retrieve(String id) {
|
||||
LruElementCache<T>.Entry entry = get(id);
|
||||
if (entry == null) {
|
||||
// not cached...
|
||||
return null;
|
||||
}
|
||||
|
||||
T element = (T) entry.getElement();
|
||||
if (element == null) {
|
||||
// gc'ed or timed out; either way...
|
||||
remove(id);
|
||||
}
|
||||
return element;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void run() {
|
||||
while (true) {
|
||||
try {
|
||||
Entry entry = (Entry) queue.remove();
|
||||
synchronized (this) {
|
||||
remove(entry.id);
|
||||
}
|
||||
} catch (InterruptedException ie) {
|
||||
throw new RuntimeException(ie);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
final class Entry extends SoftReference<T> {
|
||||
String id;
|
||||
long timeout;
|
||||
|
||||
public Entry(T element, ReferenceQueue<T> queue, int timeout) {
|
||||
super(element, queue);
|
||||
this.id = element.getId().toString();
|
||||
this.timeout = System.currentTimeMillis() + timeout;
|
||||
}
|
||||
|
||||
public T getElement() {
|
||||
if (System.currentTimeMillis() <= timeout) {
|
||||
return get();
|
||||
} else {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -14,7 +14,6 @@
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop;
|
||||
|
||||
import java.io.Closeable;
|
||||
import java.util.Iterator;
|
||||
import java.util.Map.Entry;
|
||||
|
||||
@@ -23,15 +22,17 @@ import org.apache.accumulo.core.data.Key;
|
||||
import org.apache.accumulo.core.data.Value;
|
||||
import org.apache.accumulo.core.util.PeekingIterator;
|
||||
|
||||
import com.tinkerpop.blueprints.CloseableIterable;
|
||||
import com.tinkerpop.blueprints.Element;
|
||||
|
||||
public abstract class ScannerIterable<T extends Element> implements Iterable<T>, Closeable {
|
||||
/**
|
||||
* TODO
|
||||
*/
|
||||
public abstract class ScannerIterable<T extends Element> implements CloseableIterable<T> {
|
||||
|
||||
AccumuloGraph parent;
|
||||
ScannerBase scanner;
|
||||
private ScannerBase scanner;
|
||||
|
||||
ScannerIterable(AccumuloGraph parent, ScannerBase scanner) {
|
||||
this.parent = parent;
|
||||
public ScannerIterable(ScannerBase scanner) {
|
||||
this.scanner = scanner;
|
||||
}
|
||||
|
||||
@@ -55,10 +56,10 @@ public abstract class ScannerIterable<T extends Element> implements Iterable<T>,
|
||||
close();
|
||||
}
|
||||
|
||||
class ScannerIterator implements Iterator<T> {
|
||||
PeekingIterator<Entry<Key,Value>> iterator;
|
||||
private class ScannerIterator implements Iterator<T> {
|
||||
private PeekingIterator<Entry<Key,Value>> iterator;
|
||||
|
||||
ScannerIterator(PeekingIterator<Entry<Key,Value>> iterator) {
|
||||
private ScannerIterator(PeekingIterator<Entry<Key,Value>> iterator) {
|
||||
this.iterator = iterator;
|
||||
}
|
||||
|
||||
@@ -76,7 +77,5 @@ public abstract class ScannerIterable<T extends Element> implements Iterable<T>,
|
||||
public void remove() {
|
||||
throw new UnsupportedOperationException();
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
53
src/main/java/edu/jhuapl/tinkerpop/cache/ElementCache.java
vendored
Normal file
53
src/main/java/edu/jhuapl/tinkerpop/cache/ElementCache.java
vendored
Normal file
@@ -0,0 +1,53 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop.cache;
|
||||
|
||||
import java.util.concurrent.TimeUnit;
|
||||
|
||||
import com.google.common.cache.Cache;
|
||||
import com.google.common.cache.CacheBuilder;
|
||||
import com.tinkerpop.blueprints.Element;
|
||||
|
||||
/**
|
||||
* Simple cache for retrieved graph elements,
|
||||
* backed by Guava's cache implementation.
|
||||
*/
|
||||
public class ElementCache<T extends Element> {
|
||||
|
||||
private Cache<Object, T> cache;
|
||||
|
||||
public ElementCache(int size, int timeout) {
|
||||
cache = CacheBuilder.newBuilder()
|
||||
.maximumSize(size)
|
||||
.expireAfterAccess(timeout, TimeUnit.MILLISECONDS)
|
||||
.build();
|
||||
}
|
||||
|
||||
public void cache(T element) {
|
||||
cache.put(element.getId(), element);
|
||||
}
|
||||
|
||||
public T retrieve(Object id) {
|
||||
return cache.getIfPresent(id);
|
||||
}
|
||||
|
||||
public void remove(Object id) {
|
||||
cache.invalidate(id);
|
||||
}
|
||||
|
||||
public void clear() {
|
||||
cache.invalidateAll();
|
||||
}
|
||||
}
|
||||
77
src/main/java/edu/jhuapl/tinkerpop/cache/ElementCaches.java
vendored
Normal file
77
src/main/java/edu/jhuapl/tinkerpop/cache/ElementCaches.java
vendored
Normal file
@@ -0,0 +1,77 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop.cache;
|
||||
|
||||
import com.tinkerpop.blueprints.Edge;
|
||||
import com.tinkerpop.blueprints.Element;
|
||||
import com.tinkerpop.blueprints.Vertex;
|
||||
|
||||
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration;
|
||||
import edu.jhuapl.tinkerpop.AccumuloGraphException;
|
||||
|
||||
/**
|
||||
* Utility class wrapping element caches.
|
||||
*/
|
||||
public class ElementCaches {
|
||||
private ElementCache<Vertex> vertexCache;
|
||||
private ElementCache<Edge> edgeCache;
|
||||
|
||||
public ElementCaches(AccumuloGraphConfiguration config) {
|
||||
if (config.getVertexCacheEnabled()) {
|
||||
vertexCache = new ElementCache<Vertex>(config.getVertexCacheSize(),
|
||||
config.getVertexCacheTimeout());
|
||||
}
|
||||
|
||||
if (config.getEdgeCacheEnabled()) {
|
||||
edgeCache = new ElementCache<Edge>(config.getEdgeCacheSize(),
|
||||
config.getEdgeCacheTimeout());
|
||||
}
|
||||
}
|
||||
|
||||
public <T extends Element> void cache(T element, Class<T> clazz) {
|
||||
if (pick(clazz) != null) {
|
||||
pick(clazz).cache(element);
|
||||
}
|
||||
}
|
||||
|
||||
public <T extends Element> T retrieve(Object id, Class<T> clazz) {
|
||||
return pick(clazz) != null ? pick(clazz).retrieve(id) : null;
|
||||
}
|
||||
|
||||
public <T extends Element> void remove(Object id, Class<T> clazz) {
|
||||
if (pick(clazz) != null) {
|
||||
pick(clazz).remove(id);
|
||||
}
|
||||
}
|
||||
|
||||
public <T extends Element> void clear(Class<T> clazz) {
|
||||
if (pick(clazz) != null) {
|
||||
pick(clazz).clear();
|
||||
}
|
||||
}
|
||||
|
||||
@SuppressWarnings("unchecked")
|
||||
private <T extends Element> ElementCache<T> pick(Class<T> clazz) {
|
||||
if (Vertex.class.equals(clazz)) {
|
||||
return (ElementCache<T>) vertexCache;
|
||||
}
|
||||
else if (Edge.class.equals(clazz)) {
|
||||
return (ElementCache<T>) edgeCache;
|
||||
}
|
||||
else {
|
||||
throw new AccumuloGraphException("Unknown element class: "+clazz);
|
||||
}
|
||||
}
|
||||
}
|
||||
140
src/main/java/edu/jhuapl/tinkerpop/cache/PropertyCache.java
vendored
Normal file
140
src/main/java/edu/jhuapl/tinkerpop/cache/PropertyCache.java
vendored
Normal file
@@ -0,0 +1,140 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop.cache;
|
||||
|
||||
import java.util.HashMap;
|
||||
import java.util.Map;
|
||||
import java.util.Map.Entry;
|
||||
import java.util.Set;
|
||||
|
||||
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration;
|
||||
|
||||
/**
|
||||
* Cache for storing element properties.
|
||||
* This supports a global timeout for key evictions,
|
||||
* as well as per-property eviction.
|
||||
* If caching is disabled for the given property,
|
||||
* this does nothing (stores no values).
|
||||
*/
|
||||
public class PropertyCache {
|
||||
|
||||
private final AccumuloGraphConfiguration cfg;
|
||||
private final Map<String, TimedValue> values;
|
||||
|
||||
public PropertyCache(AccumuloGraphConfiguration cfg) {
|
||||
this.cfg = cfg;
|
||||
this.values = new HashMap<String, TimedValue>();
|
||||
}
|
||||
|
||||
public boolean containsKey(String key) {
|
||||
return values.containsKey(key);
|
||||
}
|
||||
|
||||
public Set<String> keySet() {
|
||||
return values.keySet();
|
||||
}
|
||||
|
||||
public void put(String key, Object value) {
|
||||
Integer timeout = getTimeout(key);
|
||||
|
||||
// Don't cache anything without a specified timeout.
|
||||
if (timeout == null) {
|
||||
return;
|
||||
}
|
||||
values.put(key, new TimedValue(value,
|
||||
System.currentTimeMillis() + timeout));
|
||||
}
|
||||
|
||||
public void putAll(Map<String, Object> entries) {
|
||||
if(entries==null)return;
|
||||
for (Entry<String,Object> ent : entries.entrySet()) {
|
||||
put(ent.getKey(), ent.getValue());
|
||||
}
|
||||
}
|
||||
|
||||
@SuppressWarnings("unchecked")
|
||||
public <T> T get(String key) {
|
||||
long now = System.currentTimeMillis();
|
||||
|
||||
TimedValue val = values.get(key);
|
||||
if (val != null) {
|
||||
if (val.getExpiry() != null &&
|
||||
val.getExpiry() <= now) {
|
||||
remove(key);
|
||||
return null;
|
||||
}
|
||||
else {
|
||||
return (T) val.getValue();
|
||||
}
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
public void remove(String key) {
|
||||
values.remove(key);
|
||||
}
|
||||
|
||||
public void clear() {
|
||||
values.clear();
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString() {
|
||||
return values.toString();
|
||||
}
|
||||
|
||||
/**
|
||||
* Return the timeout for the given key.
|
||||
* Checks for a key-specific timeout
|
||||
* first, then the default, if any.
|
||||
* Return null if no timeout.
|
||||
* @param key
|
||||
* @return
|
||||
*/
|
||||
private Integer getTimeout(String key) {
|
||||
int timeout = cfg.getPropertyCacheTimeout(key);
|
||||
if (timeout <= 0) {
|
||||
timeout = cfg.getPropertyCacheTimeout(null);
|
||||
}
|
||||
return timeout > 0 ? timeout : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Value with associated expiry time.
|
||||
*/
|
||||
private static class TimedValue {
|
||||
private final Object value;
|
||||
private final Long expiry;
|
||||
|
||||
public TimedValue(Object value, Long expiry) {
|
||||
this.value = value;
|
||||
this.expiry = expiry;
|
||||
}
|
||||
|
||||
public Object getValue() {
|
||||
return value;
|
||||
}
|
||||
|
||||
public Long getExpiry() {
|
||||
return expiry;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString() {
|
||||
return "[" + value + ", " + expiry + "]";
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,5 +1,7 @@
|
||||
package edu.jhuapl.tinkerpop.mapreduce;
|
||||
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.io.DataInputStream;
|
||||
import java.io.IOException;
|
||||
import java.util.Iterator;
|
||||
import java.util.Map.Entry;
|
||||
@@ -24,8 +26,14 @@ import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
|
||||
import edu.jhuapl.tinkerpop.AccumuloGraph;
|
||||
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration;
|
||||
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration.InstanceType;
|
||||
import edu.jhuapl.tinkerpop.AccumuloGraphException;
|
||||
import edu.jhuapl.tinkerpop.Constants;
|
||||
|
||||
public class EdgeInputFormat extends InputFormatBase<Text,Edge> {
|
||||
|
||||
private static final String PREFIX = EdgeInputFormat.class.getSimpleName()+".";
|
||||
private static final String GRAPH_NAME = PREFIX+"graph.name";
|
||||
|
||||
static AccumuloGraphConfiguration conf;
|
||||
|
||||
@Override
|
||||
@@ -50,22 +58,21 @@ public class EdgeInputFormat extends InputFormatBase<Text,Edge> {
|
||||
|
||||
try {
|
||||
conf = new AccumuloGraphConfiguration();
|
||||
conf.setZookeeperHosts(EdgeInputFormat.getInstance(attempt).getZooKeepers());
|
||||
conf.setZooKeeperHosts(EdgeInputFormat.getInstance(attempt).getZooKeepers());
|
||||
conf.setInstanceName(EdgeInputFormat.getInstance(attempt).getInstanceName());
|
||||
conf.setUser(EdgeInputFormat.getPrincipal(attempt));
|
||||
conf.setPassword(EdgeInputFormat.getToken(attempt));
|
||||
conf.setGraphName(attempt.getConfiguration().get(AccumuloGraphConfiguration.GRAPH_NAME));
|
||||
conf.setTokenWithFallback(EdgeInputFormat.getToken(attempt));
|
||||
conf.setGraphName(attempt.getConfiguration().get(GRAPH_NAME));
|
||||
if (EdgeInputFormat.getInstance(attempt) instanceof MockInstance) {
|
||||
conf.setInstanceType(InstanceType.Mock);
|
||||
}
|
||||
parent = AccumuloGraph.open(conf.getConfiguration());
|
||||
} catch (AccumuloException e) {
|
||||
// TODO Auto-generated catch block
|
||||
e.printStackTrace();
|
||||
throw new AccumuloGraphException(e);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
|
||||
@Override
|
||||
public boolean nextKeyValue() throws IOException, InterruptedException {
|
||||
if (rowIterator.hasNext()) {
|
||||
@@ -80,17 +87,17 @@ public class EdgeInputFormat extends InputFormatBase<Text,Edge> {
|
||||
String eid = currentKey.getRow().toString();
|
||||
String colf = currentKey.getColumnFamily().toString();
|
||||
switch (colf) {
|
||||
case AccumuloGraph.SLABEL:
|
||||
case Constants.LABEL:
|
||||
currentK.set(eid);
|
||||
edge.prepareId(eid);
|
||||
String[] ids = currentKey.getColumnQualifier().toString().split(parent.IDDELIM);
|
||||
String[] ids = currentKey.getColumnQualifier().toString().split(Constants.ID_DELIM);
|
||||
edge.setSourceId(ids[1]);
|
||||
edge.setDestId(ids[0]);
|
||||
edge.setLabel(AccumuloByteSerializer.desserialize(entry.getValue().get()).toString());
|
||||
edge.setLabel(AccumuloByteSerializer.deserialize(entry.getValue().get()).toString());
|
||||
break;
|
||||
default:
|
||||
String propertyKey = currentKey.getColumnFamily().toString();
|
||||
Object propertyValue = AccumuloByteSerializer.desserialize(entry.getValue().get());
|
||||
Object propertyValue = AccumuloByteSerializer.deserialize(entry.getValue().get());
|
||||
edge.prepareProperty(propertyKey, propertyValue);
|
||||
}
|
||||
}
|
||||
@@ -105,13 +112,13 @@ public class EdgeInputFormat extends InputFormatBase<Text,Edge> {
|
||||
public static void setAccumuloGraphConfiguration(Job job, AccumuloGraphConfiguration cfg) throws AccumuloSecurityException {
|
||||
|
||||
EdgeInputFormat.setConnectorInfo(job, cfg.getUser(), new PasswordToken(cfg.getPassword()));
|
||||
EdgeInputFormat.setInputTableName(job, cfg.getEdgeTable());
|
||||
EdgeInputFormat.setInputTableName(job, cfg.getEdgeTableName());
|
||||
if (cfg.getInstanceType().equals(InstanceType.Mock)) {
|
||||
EdgeInputFormat.setMockInstance(job, cfg.getInstance());
|
||||
EdgeInputFormat.setMockInstance(job, cfg.getInstanceName());
|
||||
} else {
|
||||
EdgeInputFormat.setZooKeeperInstance(job, cfg.getInstance(), cfg.getZooKeeperHosts());
|
||||
EdgeInputFormat.setZooKeeperInstance(job, cfg.getInstanceName(), cfg.getZooKeeperHosts());
|
||||
}
|
||||
job.getConfiguration().set(AccumuloGraphConfiguration.GRAPH_NAME, cfg.getName());
|
||||
job.getConfiguration().set(GRAPH_NAME, cfg.getGraphName());
|
||||
|
||||
}
|
||||
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package edu.jhuapl.tinkerpop.mapreduce;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.Serializable;
|
||||
import java.util.Map.Entry;
|
||||
|
||||
import org.apache.accumulo.core.client.AccumuloException;
|
||||
@@ -25,10 +26,19 @@ import com.tinkerpop.blueprints.Element;
|
||||
|
||||
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
|
||||
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration;
|
||||
import edu.jhuapl.tinkerpop.AccumuloGraphException;
|
||||
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration.InstanceType;
|
||||
|
||||
public class ElementOutputFormat extends OutputFormat<NullWritable,Element> {
|
||||
|
||||
private static final String PREFIX = ElementOutputFormat.class.getSimpleName()+".";
|
||||
private static final String USER = "username";
|
||||
private static final String PASSWORD = PREFIX+"password";
|
||||
private static final String GRAPH_NAME = PREFIX+"graphName";
|
||||
private static final String INSTANCE = PREFIX+"instanceName";
|
||||
private static final String INSTANCE_TYPE = PREFIX+"instanceType";
|
||||
private static final String ZK_HOSTS = PREFIX+"zookeeperHosts";
|
||||
|
||||
@Override
|
||||
public RecordWriter<NullWritable,Element> getRecordWriter(TaskAttemptContext context) throws IOException, InterruptedException {
|
||||
return new ElementRecordWriter(context);
|
||||
@@ -43,13 +53,13 @@ public class ElementOutputFormat extends OutputFormat<NullWritable,Element> {
|
||||
acc.validate();
|
||||
Configuration jobconf = job.getConfiguration();
|
||||
|
||||
jobconf.set(AccumuloGraphConfiguration.USER, acc.getUser());
|
||||
jobconf.set(AccumuloGraphConfiguration.PASSWORD, new String(acc.getPassword().array()));
|
||||
jobconf.set(AccumuloGraphConfiguration.GRAPH_NAME, acc.getName());
|
||||
jobconf.set(AccumuloGraphConfiguration.INSTANCE, acc.getInstance());
|
||||
jobconf.set(AccumuloGraphConfiguration.INSTANCE_TYPE, acc.getInstanceType().toString());
|
||||
jobconf.set(USER, acc.getUser());
|
||||
jobconf.set(PASSWORD, acc.getPassword());
|
||||
jobconf.set(GRAPH_NAME, acc.getGraphName());
|
||||
jobconf.set(INSTANCE, acc.getInstanceName());
|
||||
jobconf.set(INSTANCE_TYPE, acc.getInstanceType().toString());
|
||||
if(acc.getInstanceType().equals(InstanceType.Distributed))
|
||||
jobconf.set(AccumuloGraphConfiguration.ZK_HOSTS, acc.getZooKeeperHosts());
|
||||
jobconf.set(ZK_HOSTS, acc.getZooKeeperHosts());
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -61,22 +71,22 @@ public class ElementOutputFormat extends OutputFormat<NullWritable,Element> {
|
||||
return new NullOutputFormat<Text,Mutation>().getOutputCommitter(context);
|
||||
}
|
||||
|
||||
class ElementRecordWriter extends RecordWriter<NullWritable,Element> {
|
||||
static class ElementRecordWriter extends RecordWriter<NullWritable,Element> implements Serializable{
|
||||
AccumuloGraphConfiguration config;
|
||||
|
||||
protected ElementRecordWriter(TaskAttemptContext context) {
|
||||
public ElementRecordWriter(TaskAttemptContext context) {
|
||||
config = new AccumuloGraphConfiguration();
|
||||
Configuration jobconf = context.getConfiguration();
|
||||
config.setUser(jobconf.get(AccumuloGraphConfiguration.USER));
|
||||
config.setPassword(jobconf.get(AccumuloGraphConfiguration.PASSWORD));
|
||||
config.setGraphName(jobconf.get(AccumuloGraphConfiguration.GRAPH_NAME));
|
||||
config.setInstanceName(jobconf.get(AccumuloGraphConfiguration.INSTANCE));
|
||||
config.setInstanceType(InstanceType.valueOf(jobconf.get(AccumuloGraphConfiguration.INSTANCE_TYPE)));
|
||||
config.setZookeeperHosts(jobconf.get(AccumuloGraphConfiguration.ZK_HOSTS));
|
||||
config.setUser(jobconf.get(USER));
|
||||
config.setPassword(jobconf.get(PASSWORD));
|
||||
config.setGraphName(jobconf.get(GRAPH_NAME));
|
||||
config.setInstanceName(jobconf.get(INSTANCE));
|
||||
config.setInstanceType(InstanceType.valueOf(jobconf.get(INSTANCE_TYPE)));
|
||||
config.setZooKeeperHosts(jobconf.get(ZK_HOSTS));
|
||||
|
||||
}
|
||||
|
||||
BatchWriter bw;
|
||||
transient BatchWriter bw;
|
||||
|
||||
@Override
|
||||
public void write(NullWritable key, Element value) throws IOException, InterruptedException {
|
||||
@@ -84,9 +94,9 @@ public class ElementOutputFormat extends OutputFormat<NullWritable,Element> {
|
||||
try {
|
||||
if (bw == null) {
|
||||
if (ele instanceof MapReduceVertex) {
|
||||
bw = config.getConnector().createBatchWriter(config.getVertexTable(), config.getBatchWriterConfig());
|
||||
bw = config.getConnector().createBatchWriter(config.getVertexTableName(), config.getBatchWriterConfig());
|
||||
} else {
|
||||
bw = config.getConnector().createBatchWriter(config.getEdgeTable(), config.getBatchWriterConfig());
|
||||
bw = config.getConnector().createBatchWriter(config.getEdgeTableName(), config.getBatchWriterConfig());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -97,9 +107,7 @@ public class ElementOutputFormat extends OutputFormat<NullWritable,Element> {
|
||||
|
||||
bw.addMutation(mut);
|
||||
} catch (TableNotFoundException | AccumuloException | AccumuloSecurityException e) {
|
||||
// TODO Auto-generated catch block
|
||||
e.printStackTrace();
|
||||
throw new RuntimeException(e);
|
||||
throw new AccumuloGraphException(e);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -109,7 +117,7 @@ public class ElementOutputFormat extends OutputFormat<NullWritable,Element> {
|
||||
try {
|
||||
bw.close();
|
||||
} catch (MutationsRejectedException e) {
|
||||
e.printStackTrace();
|
||||
throw new AccumuloGraphException(e);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -14,6 +14,8 @@
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop.mapreduce;
|
||||
|
||||
import java.io.Serializable;
|
||||
|
||||
import com.tinkerpop.blueprints.Direction;
|
||||
import com.tinkerpop.blueprints.Edge;
|
||||
import com.tinkerpop.blueprints.Vertex;
|
||||
@@ -21,7 +23,7 @@ import com.tinkerpop.blueprints.util.ExceptionFactory;
|
||||
|
||||
import edu.jhuapl.tinkerpop.AccumuloGraph;
|
||||
|
||||
public class MapReduceEdge extends MapReduceElement implements Edge {
|
||||
public class MapReduceEdge extends MapReduceElement implements Edge, Serializable {
|
||||
|
||||
String sourceId;
|
||||
String label;
|
||||
@@ -30,6 +32,8 @@ public class MapReduceEdge extends MapReduceElement implements Edge {
|
||||
MapReduceEdge(AccumuloGraph parent) {
|
||||
super(parent);
|
||||
}
|
||||
|
||||
MapReduceEdge() { super(); }
|
||||
|
||||
void setSourceId(String id) {
|
||||
sourceId = id;
|
||||
|
||||
@@ -17,7 +17,7 @@ package edu.jhuapl.tinkerpop.mapreduce;
|
||||
import java.io.DataInput;
|
||||
import java.io.DataOutput;
|
||||
import java.io.IOException;
|
||||
import java.util.Collections;
|
||||
import java.io.Serializable;
|
||||
import java.util.HashMap;
|
||||
import java.util.HashSet;
|
||||
import java.util.Map;
|
||||
@@ -32,7 +32,7 @@ import com.tinkerpop.blueprints.Graph;
|
||||
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
|
||||
import edu.jhuapl.tinkerpop.AccumuloGraph;
|
||||
|
||||
public abstract class MapReduceElement implements Element, WritableComparable<MapReduceElement> {
|
||||
public abstract class MapReduceElement implements Serializable, Element, WritableComparable<MapReduceElement> {
|
||||
|
||||
protected String id;
|
||||
|
||||
@@ -40,10 +40,14 @@ public abstract class MapReduceElement implements Element, WritableComparable<Ma
|
||||
|
||||
protected Map<String,Object> newProperties;
|
||||
|
||||
AccumuloGraph parent;
|
||||
transient AccumuloGraph parent;
|
||||
|
||||
MapReduceElement(AccumuloGraph parent) {
|
||||
this();
|
||||
this.parent = parent;
|
||||
}
|
||||
|
||||
MapReduceElement(){
|
||||
properties = new HashMap<String,Object>();
|
||||
newProperties = new HashMap<String,Object>();
|
||||
}
|
||||
@@ -65,6 +69,7 @@ public abstract class MapReduceElement implements Element, WritableComparable<Ma
|
||||
return id;
|
||||
}
|
||||
|
||||
@SuppressWarnings("unchecked")
|
||||
@Override
|
||||
public <T> T getProperty(String key) {
|
||||
|
||||
@@ -108,7 +113,7 @@ public abstract class MapReduceElement implements Element, WritableComparable<Ma
|
||||
String key = in.readUTF();
|
||||
byte[] data = new byte[in.readInt()];
|
||||
in.readFully(data);
|
||||
Object val = AccumuloByteSerializer.desserialize(data);
|
||||
Object val = AccumuloByteSerializer.deserialize(data);
|
||||
properties.put(key, val);
|
||||
}
|
||||
|
||||
@@ -117,7 +122,7 @@ public abstract class MapReduceElement implements Element, WritableComparable<Ma
|
||||
String key = in.readUTF();
|
||||
byte[] data = new byte[in.readInt()];
|
||||
in.readFully(data);
|
||||
Object val = AccumuloByteSerializer.desserialize(data);
|
||||
Object val = AccumuloByteSerializer.deserialize(data);
|
||||
newProperties.put(key, val);
|
||||
}
|
||||
|
||||
|
||||
@@ -17,6 +17,7 @@ package edu.jhuapl.tinkerpop.mapreduce;
|
||||
import java.io.DataInput;
|
||||
import java.io.DataOutput;
|
||||
import java.io.IOException;
|
||||
import java.io.Serializable;
|
||||
import java.util.Arrays;
|
||||
import java.util.Collection;
|
||||
import java.util.HashSet;
|
||||
@@ -33,7 +34,7 @@ import com.tinkerpop.blueprints.util.DefaultVertexQuery;
|
||||
|
||||
import edu.jhuapl.tinkerpop.AccumuloGraph;
|
||||
|
||||
public class MapReduceVertex extends MapReduceElement implements Vertex {
|
||||
public class MapReduceVertex extends MapReduceElement implements Vertex, Serializable {
|
||||
|
||||
List<Edge> inEdges;
|
||||
List<Edge> outEdges;
|
||||
@@ -43,8 +44,14 @@ public class MapReduceVertex extends MapReduceElement implements Vertex {
|
||||
inEdges = new LinkedList<Edge>();
|
||||
outEdges = new LinkedList<Edge>();
|
||||
}
|
||||
|
||||
MapReduceVertex(){
|
||||
//super();
|
||||
inEdges = new LinkedList<Edge>();
|
||||
outEdges = new LinkedList<Edge>();
|
||||
}
|
||||
|
||||
void prepareEdge(String id, String src, String label, String dest) {
|
||||
MapReduceEdge prepareEdge(String id, String src, String label, String dest) {
|
||||
MapReduceEdge mre = new MapReduceEdge(parent, id, src, label, dest);
|
||||
if (src.equals(getId())) {
|
||||
outEdges.add(mre);
|
||||
@@ -54,6 +61,7 @@ public class MapReduceVertex extends MapReduceElement implements Vertex {
|
||||
if (dest.equals(getId())) {
|
||||
inEdges.add(mre);
|
||||
}
|
||||
return mre;
|
||||
}
|
||||
|
||||
@Override
|
||||
|
||||
@@ -1,7 +0,0 @@
|
||||
package edu.jhuapl.tinkerpop.mapreduce;
|
||||
|
||||
import org.apache.accumulo.core.client.mapreduce.AccumuloOutputFormat;
|
||||
|
||||
public class NewElementOutputFormat extends AccumuloOutputFormat{
|
||||
|
||||
}
|
||||
@@ -1,5 +1,7 @@
|
||||
package edu.jhuapl.tinkerpop.mapreduce;
|
||||
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.io.DataInputStream;
|
||||
import java.io.IOException;
|
||||
import java.util.Iterator;
|
||||
import java.util.Map.Entry;
|
||||
@@ -12,26 +14,30 @@ import org.apache.accumulo.core.client.mock.MockInstance;
|
||||
import org.apache.accumulo.core.client.security.tokens.PasswordToken;
|
||||
import org.apache.accumulo.core.data.Key;
|
||||
import org.apache.accumulo.core.data.Value;
|
||||
import org.apache.commons.configuration.Configuration;
|
||||
import org.apache.hadoop.io.Text;
|
||||
import org.apache.hadoop.mapreduce.InputSplit;
|
||||
import org.apache.hadoop.mapreduce.Job;
|
||||
import org.apache.hadoop.mapreduce.RecordReader;
|
||||
import org.apache.hadoop.mapreduce.TaskAttemptContext;
|
||||
|
||||
import com.tinkerpop.blueprints.Graph;
|
||||
import com.tinkerpop.blueprints.Vertex;
|
||||
|
||||
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
|
||||
import edu.jhuapl.tinkerpop.AccumuloGraph;
|
||||
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration;
|
||||
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration.InstanceType;
|
||||
import edu.jhuapl.tinkerpop.AccumuloGraphException;
|
||||
import edu.jhuapl.tinkerpop.Constants;
|
||||
|
||||
public class VertexInputFormat extends InputFormatBase<Text,Vertex> {
|
||||
static AccumuloGraphConfiguration conf;
|
||||
|
||||
private static final String PREFIX = VertexInputFormat.class.getSimpleName()+".";
|
||||
private static final String GRAPH_NAME = PREFIX+"graph.name";
|
||||
|
||||
@Override
|
||||
public RecordReader<Text,Vertex> createRecordReader(InputSplit split, TaskAttemptContext attempt) throws IOException, InterruptedException {
|
||||
public RecordReader<Text,Vertex> createRecordReader(InputSplit split,
|
||||
TaskAttemptContext attempt) throws IOException, InterruptedException {
|
||||
return new VertexRecordReader();
|
||||
}
|
||||
|
||||
@@ -52,22 +58,22 @@ public class VertexInputFormat extends InputFormatBase<Text,Vertex> {
|
||||
|
||||
try {
|
||||
conf = new AccumuloGraphConfiguration();
|
||||
conf.setZookeeperHosts(VertexInputFormat.getInstance(attempt).getZooKeepers());
|
||||
conf.setZooKeeperHosts(VertexInputFormat.getInstance(attempt).getZooKeepers());
|
||||
conf.setInstanceName(VertexInputFormat.getInstance(attempt).getInstanceName());
|
||||
conf.setUser(VertexInputFormat.getPrincipal(attempt));
|
||||
conf.setPassword(VertexInputFormat.getToken(attempt));
|
||||
conf.setGraphName(attempt.getConfiguration().get(AccumuloGraphConfiguration.GRAPH_NAME));
|
||||
conf.setTokenWithFallback(VertexInputFormat.getToken(attempt));
|
||||
conf.setGraphName(attempt.getConfiguration().get(GRAPH_NAME));
|
||||
if (VertexInputFormat.getInstance(attempt) instanceof MockInstance) {
|
||||
conf.setInstanceType(InstanceType.Mock);
|
||||
}
|
||||
|
||||
parent = AccumuloGraph.open(conf.getConfiguration());
|
||||
} catch (AccumuloException e) {
|
||||
// TODO Auto-generated catch block
|
||||
e.printStackTrace();
|
||||
throw new AccumuloGraphException(e);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
|
||||
|
||||
@Override
|
||||
public boolean nextKeyValue() throws IOException, InterruptedException {
|
||||
@@ -83,23 +89,23 @@ public class VertexInputFormat extends InputFormatBase<Text,Vertex> {
|
||||
String vid = currentKey.getRow().toString();
|
||||
String colf = currentKey.getColumnFamily().toString();
|
||||
switch (colf) {
|
||||
case AccumuloGraph.SLABEL:
|
||||
case Constants.LABEL:
|
||||
currentK.set(vid);
|
||||
vertex.prepareId(vid);
|
||||
break;
|
||||
case AccumuloGraph.SINEDGE:
|
||||
String[] parts = currentKey.getColumnQualifier().toString().split(AccumuloGraph.IDDELIM);
|
||||
case Constants.IN_EDGE:
|
||||
String[] parts = currentKey.getColumnQualifier().toString().split(Constants.ID_DELIM);
|
||||
String label = new String(entry.getValue().get());
|
||||
vertex.prepareEdge(parts[1], parts[0], label, vid);
|
||||
break;
|
||||
case AccumuloGraph.SOUTEDGE:
|
||||
parts = currentKey.getColumnQualifier().toString().split(AccumuloGraph.IDDELIM);
|
||||
case Constants.OUT_EDGE:
|
||||
parts = currentKey.getColumnQualifier().toString().split(Constants.ID_DELIM);
|
||||
label = new String(entry.getValue().get());
|
||||
vertex.prepareEdge(parts[1], vid, label, parts[0]);
|
||||
break;
|
||||
default:
|
||||
String propertyKey = currentKey.getColumnFamily().toString();
|
||||
Object propertyValue = AccumuloByteSerializer.desserialize(entry.getValue().get());
|
||||
Object propertyValue = AccumuloByteSerializer.deserialize(entry.getValue().get());
|
||||
vertex.prepareProperty(propertyKey, propertyValue);
|
||||
}
|
||||
}
|
||||
@@ -114,13 +120,13 @@ public class VertexInputFormat extends InputFormatBase<Text,Vertex> {
|
||||
public static void setAccumuloGraphConfiguration(Job job, AccumuloGraphConfiguration cfg) throws AccumuloSecurityException {
|
||||
|
||||
VertexInputFormat.setConnectorInfo(job, cfg.getUser(), new PasswordToken(cfg.getPassword()));
|
||||
VertexInputFormat.setInputTableName(job, cfg.getVertexTable());
|
||||
VertexInputFormat.setInputTableName(job, cfg.getVertexTableName());
|
||||
if (cfg.getInstanceType().equals(InstanceType.Mock)) {
|
||||
VertexInputFormat.setMockInstance(job, cfg.getInstance());
|
||||
VertexInputFormat.setMockInstance(job, cfg.getInstanceName());
|
||||
} else {
|
||||
VertexInputFormat.setZooKeeperInstance(job, cfg.getInstance(), cfg.getZooKeeperHosts());
|
||||
VertexInputFormat.setZooKeeperInstance(job, cfg.getInstanceName(), cfg.getZooKeeperHosts());
|
||||
}
|
||||
job.getConfiguration().set(AccumuloGraphConfiguration.GRAPH_NAME, cfg.getName());
|
||||
job.getConfiguration().set(GRAPH_NAME, cfg.getGraphName());
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
22
src/main/java/edu/jhuapl/tinkerpop/mutator/Mutator.java
Normal file
22
src/main/java/edu/jhuapl/tinkerpop/mutator/Mutator.java
Normal file
@@ -0,0 +1,22 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop.mutator;
|
||||
|
||||
import org.apache.accumulo.core.data.Mutation;
|
||||
|
||||
public interface Mutator {
|
||||
|
||||
public Iterable<Mutation> create();
|
||||
}
|
||||
53
src/main/java/edu/jhuapl/tinkerpop/mutator/Mutators.java
Normal file
53
src/main/java/edu/jhuapl/tinkerpop/mutator/Mutators.java
Normal file
@@ -0,0 +1,53 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop.mutator;
|
||||
|
||||
import java.util.LinkedList;
|
||||
import java.util.List;
|
||||
|
||||
import org.apache.accumulo.core.client.BatchDeleter;
|
||||
import org.apache.accumulo.core.client.BatchWriter;
|
||||
import org.apache.accumulo.core.client.MutationsRejectedException;
|
||||
import org.apache.accumulo.core.data.Range;
|
||||
|
||||
import com.tinkerpop.blueprints.Element;
|
||||
|
||||
import edu.jhuapl.tinkerpop.AccumuloGraphException;
|
||||
|
||||
public class Mutators {
|
||||
|
||||
public static void apply(BatchWriter writer, Mutator mut) {
|
||||
try {
|
||||
writer.addMutations(mut.create());
|
||||
} catch (MutationsRejectedException e) {
|
||||
throw new AccumuloGraphException(e);
|
||||
}
|
||||
}
|
||||
|
||||
public static void deleteElementRanges(BatchDeleter deleter, Element... elements) {
|
||||
List<Range> ranges = new LinkedList<Range>();
|
||||
|
||||
for (Element element : elements) {
|
||||
ranges.add(new Range(element.getId().toString()));
|
||||
}
|
||||
deleter.setRanges(ranges);
|
||||
|
||||
try {
|
||||
deleter.delete();
|
||||
} catch (Exception e) {
|
||||
throw new AccumuloGraphException(e);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop.mutator.edge;
|
||||
|
||||
import com.tinkerpop.blueprints.Direction;
|
||||
import com.tinkerpop.blueprints.Edge;
|
||||
|
||||
import edu.jhuapl.tinkerpop.mutator.Mutator;
|
||||
|
||||
public abstract class BaseEdgeMutator implements Mutator {
|
||||
|
||||
protected final String id;
|
||||
protected final String outVertexId;
|
||||
protected final String inVertexId;
|
||||
protected final String label;
|
||||
|
||||
public BaseEdgeMutator(Edge edge) {
|
||||
this(edge.getId().toString(),
|
||||
edge.getVertex(Direction.OUT).getId().toString(),
|
||||
edge.getVertex(Direction.IN).getId().toString(),
|
||||
edge.getLabel());
|
||||
}
|
||||
public BaseEdgeMutator(String id, String outVertexId, String inVertexId, String label) {
|
||||
this.id = id;
|
||||
this.outVertexId = outVertexId;
|
||||
this.inVertexId = inVertexId;
|
||||
this.label = label;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,78 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop.mutator.edge;
|
||||
|
||||
import org.apache.accumulo.core.data.Mutation;
|
||||
|
||||
import com.google.common.collect.Lists;
|
||||
import com.tinkerpop.blueprints.Edge;
|
||||
|
||||
import edu.jhuapl.tinkerpop.Constants;
|
||||
|
||||
public class EdgeEndpointsMutator {
|
||||
|
||||
private EdgeEndpointsMutator() {
|
||||
}
|
||||
|
||||
public static class Add extends BaseEdgeMutator {
|
||||
|
||||
public Add(Edge edge) {
|
||||
super(edge);
|
||||
}
|
||||
|
||||
public Add(String id, String outVertexId, String inVertexId, String label) {
|
||||
super(id, outVertexId, inVertexId, label);
|
||||
}
|
||||
|
||||
@Override
|
||||
public Iterable<Mutation> create() {
|
||||
Mutation in = new Mutation(inVertexId);
|
||||
in.put(Constants.IN_EDGE.getBytes(),
|
||||
(outVertexId + Constants.ID_DELIM + id).getBytes(),
|
||||
(Constants.ID_DELIM + label).getBytes());
|
||||
|
||||
Mutation out = new Mutation(outVertexId);
|
||||
out.put(Constants.OUT_EDGE.getBytes(),
|
||||
(inVertexId + Constants.ID_DELIM + id).getBytes(),
|
||||
(Constants.ID_DELIM + label).getBytes());
|
||||
|
||||
return Lists.newArrayList(in, out);
|
||||
}
|
||||
}
|
||||
|
||||
public static class Delete extends BaseEdgeMutator {
|
||||
|
||||
public Delete(Edge edge) {
|
||||
super(edge);
|
||||
}
|
||||
|
||||
public Delete(String id, String outVertexId, String inVertexId, String label) {
|
||||
super(id, outVertexId, inVertexId, label);
|
||||
}
|
||||
|
||||
@Override
|
||||
public Iterable<Mutation> create() {
|
||||
Mutation in = new Mutation(inVertexId);
|
||||
in.putDelete(Constants.IN_EDGE.getBytes(),
|
||||
(outVertexId + Constants.ID_DELIM + id).getBytes());
|
||||
|
||||
Mutation out = new Mutation(outVertexId);
|
||||
out.putDelete(Constants.OUT_EDGE.getBytes(),
|
||||
(inVertexId + Constants.ID_DELIM + id).getBytes());
|
||||
|
||||
return Lists.newArrayList(in, out);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,67 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop.mutator.edge;
|
||||
|
||||
import org.apache.accumulo.core.data.Mutation;
|
||||
|
||||
import com.google.common.collect.Lists;
|
||||
import com.tinkerpop.blueprints.Edge;
|
||||
|
||||
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
|
||||
import edu.jhuapl.tinkerpop.Constants;
|
||||
|
||||
public final class EdgeMutator {
|
||||
|
||||
public static class Add extends BaseEdgeMutator {
|
||||
|
||||
public Add(Edge edge) {
|
||||
super(edge);
|
||||
}
|
||||
|
||||
public Add(String id, String outVertexId, String inVertexId, String label) {
|
||||
super(id, outVertexId, inVertexId, label);
|
||||
}
|
||||
|
||||
@Override
|
||||
public Iterable<Mutation> create() {
|
||||
Mutation m = new Mutation(id);
|
||||
m.put(Constants.LABEL.getBytes(),
|
||||
(inVertexId + Constants.ID_DELIM + outVertexId).getBytes(),
|
||||
AccumuloByteSerializer.serialize(label));
|
||||
|
||||
return Lists.newArrayList(m);
|
||||
}
|
||||
}
|
||||
|
||||
public static class Delete extends BaseEdgeMutator {
|
||||
|
||||
public Delete(Edge edge) {
|
||||
super(edge);
|
||||
}
|
||||
|
||||
public Delete(String id, String outVertexId, String inVertexId, String label) {
|
||||
super(id, outVertexId, inVertexId, label);
|
||||
}
|
||||
|
||||
@Override
|
||||
public Iterable<Mutation> create() {
|
||||
Mutation m = new Mutation(id);
|
||||
m.putDelete(Constants.LABEL.getBytes(),
|
||||
(inVertexId + Constants.ID_DELIM + outVertexId).getBytes());
|
||||
|
||||
return Lists.newArrayList(m);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,76 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop.mutator.index;
|
||||
|
||||
import org.apache.accumulo.core.data.Mutation;
|
||||
|
||||
import com.google.common.collect.Lists;
|
||||
import com.tinkerpop.blueprints.Element;
|
||||
|
||||
import edu.jhuapl.tinkerpop.Constants;
|
||||
import edu.jhuapl.tinkerpop.Constants.IndexMetadataEntryType;
|
||||
import edu.jhuapl.tinkerpop.mutator.Mutator;
|
||||
|
||||
/**
|
||||
* Mutators for index metadata table entries.
|
||||
*/
|
||||
public class IndexMetadataMutator {
|
||||
|
||||
private IndexMetadataMutator() { }
|
||||
|
||||
public static class Add implements Mutator {
|
||||
|
||||
private final String key;
|
||||
private final Class<? extends Element> elementClass;
|
||||
private final IndexMetadataEntryType entryType;
|
||||
|
||||
public Add(String key, Class<? extends Element> elementClass,
|
||||
IndexMetadataEntryType entryType) {
|
||||
this.key = key;
|
||||
this.elementClass = elementClass;
|
||||
this.entryType = entryType;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Iterable<Mutation> create() {
|
||||
Mutation m = new Mutation(key);
|
||||
m.put(entryType.name().getBytes(),
|
||||
elementClass.getName().getBytes(), Constants.EMPTY);
|
||||
return Lists.newArrayList(m);
|
||||
}
|
||||
}
|
||||
|
||||
public static class Delete implements Mutator {
|
||||
|
||||
private final String key;
|
||||
private final Class<? extends Element> elementClass;
|
||||
private final IndexMetadataEntryType entryType;
|
||||
|
||||
public Delete(String indexName, Class<? extends Element> elementClass,
|
||||
IndexMetadataEntryType entryType) {
|
||||
this.key = indexName;
|
||||
this.elementClass = elementClass;
|
||||
this.entryType = entryType;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Iterable<Mutation> create() {
|
||||
Mutation m = new Mutation(key);
|
||||
m.putDelete(entryType.name().getBytes(),
|
||||
elementClass.getName().getBytes());
|
||||
return Lists.newArrayList(m);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,75 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop.mutator.index;
|
||||
|
||||
import org.apache.accumulo.core.data.Mutation;
|
||||
|
||||
import com.google.common.collect.Lists;
|
||||
import com.tinkerpop.blueprints.Element;
|
||||
|
||||
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
|
||||
import edu.jhuapl.tinkerpop.Constants;
|
||||
import edu.jhuapl.tinkerpop.mutator.Mutator;
|
||||
|
||||
/**
|
||||
* Mutators for vertex/edge index tables.
|
||||
*/
|
||||
public class IndexValueMutator {
|
||||
|
||||
private IndexValueMutator() { }
|
||||
|
||||
public static class Add implements Mutator {
|
||||
|
||||
private final Element element;
|
||||
private final String key;
|
||||
private final Object value;
|
||||
|
||||
public Add(Element element, String key, Object value) {
|
||||
this.element = element;
|
||||
this.key = key;
|
||||
this.value = value;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Iterable<Mutation> create() {
|
||||
byte[] bytes = AccumuloByteSerializer.serialize(value);
|
||||
Mutation m = new Mutation(bytes);
|
||||
m.put(key.getBytes(), element.getId().toString()
|
||||
.getBytes(), Constants.EMPTY);
|
||||
return Lists.newArrayList(m);
|
||||
}
|
||||
}
|
||||
|
||||
public static class Delete implements Mutator {
|
||||
|
||||
private final Element element;
|
||||
private final String key;
|
||||
private final Object value;
|
||||
|
||||
public Delete(Element element, String key, Object value) {
|
||||
this.element = element;
|
||||
this.key = key;
|
||||
this.value = value;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Iterable<Mutation> create() {
|
||||
byte[] bytes = AccumuloByteSerializer.serialize(value);
|
||||
Mutation m = new Mutation(bytes);
|
||||
m.putDelete(key, element.getId().toString());
|
||||
return Lists.newArrayList(m);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop.mutator.property;
|
||||
|
||||
import edu.jhuapl.tinkerpop.mutator.Mutator;
|
||||
|
||||
public abstract class BasePropertyMutator implements Mutator {
|
||||
|
||||
protected final String id;
|
||||
protected final String key;
|
||||
|
||||
public BasePropertyMutator(String id, String key) {
|
||||
this.id = id;
|
||||
this.key = key;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop.mutator.property;
|
||||
|
||||
import org.apache.accumulo.core.data.Mutation;
|
||||
|
||||
import com.google.common.collect.Lists;
|
||||
import edu.jhuapl.tinkerpop.Constants;
|
||||
|
||||
public class ClearPropertyMutator extends BasePropertyMutator {
|
||||
|
||||
public ClearPropertyMutator(String id, String key) {
|
||||
super(id, key);
|
||||
}
|
||||
|
||||
@Override
|
||||
public Iterable<Mutation> create() {
|
||||
Mutation m = new Mutation(id);
|
||||
m.putDelete(key.getBytes(), Constants.EMPTY);
|
||||
return Lists.newArrayList(m);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop.mutator.property;
|
||||
|
||||
import org.apache.accumulo.core.data.Mutation;
|
||||
|
||||
import com.google.common.collect.Lists;
|
||||
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
|
||||
import edu.jhuapl.tinkerpop.Constants;
|
||||
|
||||
public class WritePropertyMutator extends BasePropertyMutator {
|
||||
|
||||
private final Object value;
|
||||
|
||||
public WritePropertyMutator(String id, String key, Object value) {
|
||||
super(id, key);
|
||||
this.value = value;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Iterable<Mutation> create() {
|
||||
byte[] bytes = AccumuloByteSerializer.serialize(value);
|
||||
Mutation m = new Mutation(id);
|
||||
m.put(key.getBytes(), Constants.EMPTY, bytes);
|
||||
return Lists.newArrayList(m);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop.mutator.vertex;
|
||||
|
||||
import org.apache.accumulo.core.data.Mutation;
|
||||
|
||||
import com.google.common.collect.Lists;
|
||||
import edu.jhuapl.tinkerpop.Constants;
|
||||
import edu.jhuapl.tinkerpop.mutator.Mutator;
|
||||
|
||||
public final class AddVertexMutator implements Mutator {
|
||||
|
||||
private final String id;
|
||||
|
||||
public AddVertexMutator(String id) {
|
||||
this.id = id;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Iterable<Mutation> create() {
|
||||
Mutation m = new Mutation(id);
|
||||
m.put(Constants.LABEL.getBytes(),
|
||||
Constants.EXISTS.getBytes(), Constants.EMPTY);
|
||||
return Lists.newArrayList(m);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop.parser;
|
||||
|
||||
import edu.jhuapl.tinkerpop.AccumuloEdge;
|
||||
import edu.jhuapl.tinkerpop.GlobalInstances;
|
||||
|
||||
/**
|
||||
* Edge-specific index parser.
|
||||
*/
|
||||
public class EdgeIndexParser extends ElementIndexParser<AccumuloEdge> {
|
||||
|
||||
public EdgeIndexParser(GlobalInstances globals) {
|
||||
super(globals);
|
||||
}
|
||||
|
||||
@Override
|
||||
protected AccumuloEdge instantiate(String id) {
|
||||
return new AccumuloEdge(globals, id);
|
||||
}
|
||||
}
|
||||
70
src/main/java/edu/jhuapl/tinkerpop/parser/EdgeParser.java
Normal file
70
src/main/java/edu/jhuapl/tinkerpop/parser/EdgeParser.java
Normal file
@@ -0,0 +1,70 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop.parser;
|
||||
|
||||
import java.util.Map.Entry;
|
||||
|
||||
import org.apache.accumulo.core.data.Key;
|
||||
import org.apache.accumulo.core.data.Value;
|
||||
|
||||
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
|
||||
import edu.jhuapl.tinkerpop.AccumuloEdge;
|
||||
import edu.jhuapl.tinkerpop.AccumuloGraphException;
|
||||
import edu.jhuapl.tinkerpop.AccumuloVertex;
|
||||
import edu.jhuapl.tinkerpop.Constants;
|
||||
import edu.jhuapl.tinkerpop.GlobalInstances;
|
||||
|
||||
/**
|
||||
* TODO
|
||||
*/
|
||||
public class EdgeParser extends ElementParser<AccumuloEdge> {
|
||||
|
||||
public EdgeParser(GlobalInstances globals) {
|
||||
super(globals);
|
||||
}
|
||||
|
||||
@Override
|
||||
public AccumuloEdge parse(String id, Iterable<Entry<Key,Value>> entries) {
|
||||
AccumuloEdge edge = makeEdge(id, entries);
|
||||
setInMemoryProperties(edge, entries);
|
||||
return edge;
|
||||
}
|
||||
|
||||
/**
|
||||
* Make and return an edge object. If the entries
|
||||
* contain label/endpoint information, set those too.
|
||||
* @param id
|
||||
* @param entries
|
||||
* @return
|
||||
*/
|
||||
private AccumuloEdge makeEdge(String id, Iterable<Entry<Key,Value>> entries) {
|
||||
for (Entry<Key, Value> entry : entries) {
|
||||
String cf = entry.getKey().getColumnFamily().toString();
|
||||
if (Constants.LABEL.equals(cf)) {
|
||||
String cq = entry.getKey().getColumnQualifier().toString();
|
||||
String[] parts = cq.split(Constants.ID_DELIM);
|
||||
String inVertexId = parts[0];
|
||||
String outVertexId = parts[1];
|
||||
String label = AccumuloByteSerializer.deserialize(entry.getValue().get());
|
||||
return new AccumuloEdge(globals, id,
|
||||
new AccumuloVertex(globals, inVertexId),
|
||||
new AccumuloVertex(globals, outVertexId), label);
|
||||
}
|
||||
}
|
||||
|
||||
// This should not happen.
|
||||
throw new AccumuloGraphException("Unable to parse edge from entries");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,72 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop.parser;
|
||||
|
||||
import java.util.Iterator;
|
||||
import java.util.Map.Entry;
|
||||
|
||||
import org.apache.accumulo.core.data.Key;
|
||||
import org.apache.accumulo.core.data.Value;
|
||||
|
||||
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
|
||||
import edu.jhuapl.tinkerpop.AccumuloElement;
|
||||
import edu.jhuapl.tinkerpop.AccumuloGraphException;
|
||||
import edu.jhuapl.tinkerpop.GlobalInstances;
|
||||
|
||||
/**
|
||||
* Parser for elements based on an index table.
|
||||
*/
|
||||
public abstract class ElementIndexParser<T extends AccumuloElement>
|
||||
implements EntryParser<T> {
|
||||
|
||||
protected final GlobalInstances globals;
|
||||
|
||||
public ElementIndexParser(GlobalInstances globals) {
|
||||
this.globals = globals;
|
||||
}
|
||||
|
||||
@Override
|
||||
public T parse(Iterable<Entry<Key, Value>> entries) {
|
||||
Iterator<Entry<Key, Value>> it = entries.iterator();
|
||||
|
||||
if (it.hasNext()) {
|
||||
Entry<Key, Value> entry = it.next();
|
||||
|
||||
if (it.hasNext()) {
|
||||
throw new AccumuloGraphException("Unexpected multiple entries for index table");
|
||||
}
|
||||
|
||||
String id = entry.getKey().getColumnQualifierData().toString();
|
||||
T element = instantiate(id);
|
||||
|
||||
// While we're here, read the property key/value.
|
||||
String key = entry.getKey().getColumnFamily().toString();
|
||||
Object value = AccumuloByteSerializer.deserialize(entry.getKey().getRow().getBytes());
|
||||
element.setPropertyInMemory(key, value);
|
||||
|
||||
return element;
|
||||
}
|
||||
else {
|
||||
throw new AccumuloGraphException("No index table entries found");
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Instantiate an object to be returned.
|
||||
* @param id
|
||||
* @return
|
||||
*/
|
||||
protected abstract T instantiate(String id);
|
||||
}
|
||||
65
src/main/java/edu/jhuapl/tinkerpop/parser/ElementParser.java
Normal file
65
src/main/java/edu/jhuapl/tinkerpop/parser/ElementParser.java
Normal file
@@ -0,0 +1,65 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop.parser;
|
||||
|
||||
import java.util.Map;
|
||||
import java.util.Map.Entry;
|
||||
|
||||
import org.apache.accumulo.core.data.Key;
|
||||
import org.apache.accumulo.core.data.Value;
|
||||
|
||||
import edu.jhuapl.tinkerpop.AccumuloElement;
|
||||
import edu.jhuapl.tinkerpop.GlobalInstances;
|
||||
|
||||
/**
|
||||
* TODO
|
||||
*/
|
||||
public abstract class ElementParser<T extends AccumuloElement> implements EntryParser<T> {
|
||||
|
||||
protected GlobalInstances globals;
|
||||
|
||||
public ElementParser(GlobalInstances globals) {
|
||||
this.globals = globals;
|
||||
}
|
||||
|
||||
@Override
|
||||
public T parse(Iterable<Entry<Key, Value>> entries) {
|
||||
String id = entries.iterator().next().getKey().getRow().toString();
|
||||
return parse(id, entries);
|
||||
}
|
||||
|
||||
/**
|
||||
* Given the element id and set of entries,
|
||||
* create the type of element. This can leverage
|
||||
* the property loader, etc.
|
||||
* @param id
|
||||
* @param entries
|
||||
* @return
|
||||
*/
|
||||
public abstract T parse(String id, Iterable<Entry<Key, Value>> entries);
|
||||
|
||||
/**
|
||||
* Parse out the property entries and set them for the given element.
|
||||
* @param element
|
||||
* @param entries
|
||||
*/
|
||||
protected void setInMemoryProperties(T element, Iterable<Entry<Key, Value>> entries) {
|
||||
Map<String, Object> props = new PropertyParser().parse(entries);
|
||||
if(props == null) return;
|
||||
for (Entry<String, Object> ent : props.entrySet()) {
|
||||
element.setPropertyInMemory(ent.getKey(), ent.getValue());
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -12,30 +12,17 @@
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop;
|
||||
package edu.jhuapl.tinkerpop.parser;
|
||||
|
||||
import java.util.Map.Entry;
|
||||
|
||||
import org.apache.accumulo.core.data.Key;
|
||||
import org.apache.accumulo.core.data.Value;
|
||||
|
||||
public class DataConverter {
|
||||
/**
|
||||
* TODO
|
||||
*/
|
||||
public interface EntryParser<T> {
|
||||
|
||||
public static Value toValue(Object data) {
|
||||
if (data instanceof String) {
|
||||
String val = "S" + data;
|
||||
return new Value(val.getBytes());
|
||||
} else if (data instanceof Value) {
|
||||
return (Value) data;
|
||||
}
|
||||
|
||||
throw new UnsupportedOperationException("TODO: handle " + data.getClass());
|
||||
}
|
||||
|
||||
@SuppressWarnings("unchecked")
|
||||
public static <T> T fromValue(Value value) {
|
||||
byte[] data = value.get();
|
||||
switch (data[0]) {
|
||||
case 'S':
|
||||
return (T) new String(data, 1, data.length - 1);
|
||||
}
|
||||
throw new UnsupportedOperationException("TODO: handle type: " + data[0]);
|
||||
}
|
||||
public T parse(Iterable<Entry<Key, Value>> entries);
|
||||
}
|
||||
44
src/main/java/edu/jhuapl/tinkerpop/parser/IndexedItem.java
Normal file
44
src/main/java/edu/jhuapl/tinkerpop/parser/IndexedItem.java
Normal file
@@ -0,0 +1,44 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop.parser;
|
||||
|
||||
import com.tinkerpop.blueprints.Element;
|
||||
import com.tinkerpop.blueprints.IndexableGraph;
|
||||
import com.tinkerpop.blueprints.KeyIndexableGraph;
|
||||
|
||||
/**
|
||||
* An indexed item. For {@link IndexableGraph},
|
||||
* the key is the index name. For {@link KeyIndexableGraph}
|
||||
* the key is the indexed key.
|
||||
* @author Michael Lieberman
|
||||
*
|
||||
*/
|
||||
public class IndexedItem {
|
||||
private final String key;
|
||||
private final Class<? extends Element> elementClass;
|
||||
|
||||
public IndexedItem(String key, Class<? extends Element> elementClass) {
|
||||
this.key = key;
|
||||
this.elementClass = elementClass;
|
||||
}
|
||||
|
||||
public String getKey() {
|
||||
return key;
|
||||
}
|
||||
|
||||
public Class<? extends Element> getElementClass() {
|
||||
return elementClass;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop.parser;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Map.Entry;
|
||||
|
||||
import org.apache.accumulo.core.data.Key;
|
||||
import org.apache.accumulo.core.data.Value;
|
||||
|
||||
import com.tinkerpop.blueprints.Edge;
|
||||
import com.tinkerpop.blueprints.Element;
|
||||
import com.tinkerpop.blueprints.IndexableGraph;
|
||||
import com.tinkerpop.blueprints.KeyIndexableGraph;
|
||||
import com.tinkerpop.blueprints.Vertex;
|
||||
|
||||
import edu.jhuapl.tinkerpop.AccumuloGraphException;
|
||||
|
||||
/**
|
||||
* Entry parser for index metadata. The format
|
||||
* is the same for both {@link IndexableGraph}
|
||||
* and {@link KeyIndexableGraph} functionality.
|
||||
* For the former, this parser returns names of
|
||||
* indexes. For the latter, the parser returns
|
||||
* indexed keys.
|
||||
*/
|
||||
public class IndexedItemsListParser implements EntryParser<List<IndexedItem>> {
|
||||
|
||||
private final Class<? extends Element> elementClass;
|
||||
|
||||
/**
|
||||
* Constructor to return all items regardless
|
||||
* of element class.
|
||||
*/
|
||||
public IndexedItemsListParser() {
|
||||
this(Element.class);
|
||||
}
|
||||
|
||||
/**
|
||||
* Create a parser for items of the specified element class.
|
||||
* This may be Vertex, Edge, or Element.
|
||||
* @param elementClass
|
||||
*/
|
||||
public IndexedItemsListParser(Class<? extends Element> elementClass) {
|
||||
// Validate element class.
|
||||
if (!Vertex.class.equals(elementClass) &&
|
||||
!Edge.class.equals(elementClass) &&
|
||||
!Element.class.equals(elementClass)) {
|
||||
throw new IllegalArgumentException("elementClass must be Vertex, Edge or Element");
|
||||
}
|
||||
this.elementClass = elementClass;
|
||||
}
|
||||
|
||||
@SuppressWarnings("unchecked")
|
||||
@Override
|
||||
public List<IndexedItem> parse(Iterable<Entry<Key,Value>> entries) {
|
||||
List<IndexedItem> items = new ArrayList<IndexedItem>();
|
||||
|
||||
for (Entry<Key, Value> entry : entries) {
|
||||
Class<? extends Element> clazz;
|
||||
try {
|
||||
clazz = (Class<? extends Element>) Class.forName(entry.getKey()
|
||||
.getColumnQualifier().toString());
|
||||
} catch (ClassNotFoundException e) {
|
||||
throw new AccumuloGraphException(e);
|
||||
}
|
||||
|
||||
if (Element.class.equals(elementClass) ||
|
||||
elementClass.equals(clazz)) {
|
||||
IndexedItem item = new IndexedItem(entry.getKey()
|
||||
.getRow().toString(), clazz);
|
||||
items.add(item);
|
||||
}
|
||||
}
|
||||
|
||||
return items;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop.parser;
|
||||
|
||||
import java.util.HashMap;
|
||||
import java.util.Map;
|
||||
import java.util.Map.Entry;
|
||||
|
||||
import org.apache.accumulo.core.data.Key;
|
||||
import org.apache.accumulo.core.data.Value;
|
||||
|
||||
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
|
||||
import edu.jhuapl.tinkerpop.Constants;
|
||||
|
||||
/**
|
||||
* TODO
|
||||
*/
|
||||
public class PropertyParser implements EntryParser<Map<String, Object>> {
|
||||
|
||||
@Override
|
||||
public Map<String,Object> parse(Iterable<Entry<Key,Value>> entries) {
|
||||
Map<String, Object> props = null;
|
||||
|
||||
for (Entry<Key, Value> entry : entries) {
|
||||
if (props == null) {
|
||||
props = new HashMap<String, Object>();
|
||||
}
|
||||
|
||||
Key key = entry.getKey();
|
||||
|
||||
if (!isMetaKey(key)) {
|
||||
String attr = key.getColumnFamily().toString();
|
||||
Object value = AccumuloByteSerializer.deserialize(entry.getValue().get());
|
||||
props.put(attr, value);
|
||||
}
|
||||
}
|
||||
|
||||
return props;
|
||||
}
|
||||
|
||||
/**
|
||||
* Test whether the given Accumulo key represents a
|
||||
* metadata key (e.g. existence, edge endpoint, etc),
|
||||
* rather than a property.
|
||||
* @param key
|
||||
* @return
|
||||
*/
|
||||
private static boolean isMetaKey(Key key) {
|
||||
String cf = key.getColumnFamily().toString();
|
||||
return Constants.LABEL.equals(cf) ||
|
||||
Constants.IN_EDGE.equals(cf) ||
|
||||
Constants.OUT_EDGE.equals(cf);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop.parser;
|
||||
|
||||
import edu.jhuapl.tinkerpop.AccumuloVertex;
|
||||
import edu.jhuapl.tinkerpop.GlobalInstances;
|
||||
|
||||
/**
|
||||
* Vertex-specific index parser.
|
||||
*/
|
||||
public class VertexIndexParser extends ElementIndexParser<AccumuloVertex> {
|
||||
|
||||
public VertexIndexParser(GlobalInstances globals) {
|
||||
super(globals);
|
||||
}
|
||||
|
||||
@Override
|
||||
protected AccumuloVertex instantiate(String id) {
|
||||
return new AccumuloVertex(globals, id);
|
||||
}
|
||||
}
|
||||
@@ -12,29 +12,29 @@
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop;
|
||||
package edu.jhuapl.tinkerpop.parser;
|
||||
|
||||
import java.util.Map.Entry;
|
||||
|
||||
import org.apache.accumulo.core.data.Key;
|
||||
import org.apache.accumulo.core.data.Value;
|
||||
|
||||
public enum EntryLocation {
|
||||
import edu.jhuapl.tinkerpop.AccumuloVertex;
|
||||
import edu.jhuapl.tinkerpop.GlobalInstances;
|
||||
|
||||
Row, ColF, ColQ, Value;
|
||||
/**
|
||||
* TODO
|
||||
*/
|
||||
public class VertexParser extends ElementParser<AccumuloVertex> {
|
||||
|
||||
public String extract(Entry<Key,Value> entry) {
|
||||
switch (this) {
|
||||
case Row:
|
||||
return entry.getKey().getRow().toString();
|
||||
case ColF:
|
||||
return entry.getKey().getColumnFamily().toString();
|
||||
case ColQ:
|
||||
return entry.getKey().getColumnQualifier().toString();
|
||||
case Value:
|
||||
return new String(entry.getValue().get());
|
||||
default:
|
||||
throw new AccumuloGraphException("Unexpected type: " + this);
|
||||
}
|
||||
public VertexParser(GlobalInstances globals) {
|
||||
super(globals);
|
||||
}
|
||||
|
||||
@Override
|
||||
public AccumuloVertex parse(String id, Iterable<Entry<Key,Value>> entries) {
|
||||
AccumuloVertex vertex = new AccumuloVertex(globals, id);
|
||||
setInMemoryProperties(vertex, entries);
|
||||
return vertex;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop.tables;
|
||||
|
||||
import java.util.Collections;
|
||||
import java.util.Map.Entry;
|
||||
|
||||
import org.apache.accumulo.core.client.BatchDeleter;
|
||||
import org.apache.accumulo.core.client.BatchScanner;
|
||||
import org.apache.accumulo.core.client.BatchWriter;
|
||||
import org.apache.accumulo.core.client.Scanner;
|
||||
import org.apache.accumulo.core.data.Key;
|
||||
import org.apache.accumulo.core.data.Range;
|
||||
import org.apache.accumulo.core.data.Value;
|
||||
|
||||
import edu.jhuapl.tinkerpop.AccumuloGraphException;
|
||||
import edu.jhuapl.tinkerpop.GlobalInstances;
|
||||
|
||||
/**
|
||||
* Table wrapper with common functionality.
|
||||
*/
|
||||
public abstract class BaseTableWrapper {
|
||||
|
||||
protected GlobalInstances globals;
|
||||
private String tableName;
|
||||
|
||||
public BaseTableWrapper(GlobalInstances globals, String tableName) {
|
||||
this.globals = globals;
|
||||
this.tableName = tableName;
|
||||
}
|
||||
|
||||
protected Scanner getScanner() {
|
||||
try {
|
||||
return globals.getConfig().getConnector().createScanner(tableName,
|
||||
globals.getConfig().getAuthorizations());
|
||||
|
||||
} catch (Exception e) {
|
||||
throw new AccumuloGraphException(e);
|
||||
}
|
||||
}
|
||||
|
||||
protected BatchScanner getBatchScanner() {
|
||||
try {
|
||||
BatchScanner scanner = globals.getConfig().getConnector().createBatchScanner(tableName,
|
||||
globals.getConfig().getAuthorizations(), globals.getConfig().getQueryThreads());
|
||||
scanner.setRanges(Collections.singletonList(new Range()));
|
||||
return scanner;
|
||||
} catch (Exception e) {
|
||||
throw new AccumuloGraphException(e);
|
||||
}
|
||||
}
|
||||
|
||||
protected BatchWriter getWriter() {
|
||||
try {
|
||||
return globals.getMtbw().getBatchWriter(tableName);
|
||||
} catch (Exception e) {
|
||||
throw new AccumuloGraphException(e);
|
||||
}
|
||||
}
|
||||
|
||||
protected BatchDeleter getDeleter() {
|
||||
try {
|
||||
return globals.getConfig().getConnector().createBatchDeleter(tableName,
|
||||
globals.getConfig().getAuthorizations(), globals.getConfig().getMaxWriteThreads(),
|
||||
globals.getConfig().getBatchWriterConfig());
|
||||
} catch (Exception e) {
|
||||
throw new AccumuloGraphException(e);
|
||||
}
|
||||
}
|
||||
|
||||
public void dump() {
|
||||
System.out.println("Dump of table "+tableName+":");
|
||||
Scanner s = getScanner();
|
||||
for (Entry<Key, Value> entry : s) {
|
||||
System.out.println(" "+entry);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,177 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop.tables.core;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.Iterator;
|
||||
import java.util.List;
|
||||
import java.util.Map.Entry;
|
||||
import java.util.regex.Pattern;
|
||||
|
||||
import org.apache.accumulo.core.client.BatchScanner;
|
||||
import org.apache.accumulo.core.client.IteratorSetting;
|
||||
import org.apache.accumulo.core.client.Scanner;
|
||||
import org.apache.accumulo.core.data.Key;
|
||||
import org.apache.accumulo.core.data.Range;
|
||||
import org.apache.accumulo.core.data.Value;
|
||||
import org.apache.accumulo.core.iterators.user.RegExFilter;
|
||||
import org.apache.accumulo.core.util.PeekingIterator;
|
||||
import org.apache.hadoop.io.Text;
|
||||
|
||||
import com.tinkerpop.blueprints.CloseableIterable;
|
||||
import com.tinkerpop.blueprints.Edge;
|
||||
|
||||
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
|
||||
import edu.jhuapl.tinkerpop.AccumuloEdge;
|
||||
import edu.jhuapl.tinkerpop.AccumuloGraphException;
|
||||
import edu.jhuapl.tinkerpop.AccumuloGraphUtils;
|
||||
import edu.jhuapl.tinkerpop.AccumuloVertex;
|
||||
import edu.jhuapl.tinkerpop.Constants;
|
||||
import edu.jhuapl.tinkerpop.GlobalInstances;
|
||||
import edu.jhuapl.tinkerpop.ScannerIterable;
|
||||
import edu.jhuapl.tinkerpop.mutator.Mutators;
|
||||
import edu.jhuapl.tinkerpop.mutator.edge.EdgeMutator;
|
||||
import edu.jhuapl.tinkerpop.parser.EdgeParser;
|
||||
|
||||
|
||||
/**
|
||||
* Wrapper around {@link Edge} tables.
|
||||
*/
|
||||
public class EdgeTableWrapper extends ElementTableWrapper {
|
||||
|
||||
public EdgeTableWrapper(GlobalInstances globals) {
|
||||
super(globals, globals.getConfig().getEdgeTableName());
|
||||
}
|
||||
|
||||
/**
|
||||
* Write the given edge to the edge table. Does not
|
||||
* currently write the edge's properties.
|
||||
*
|
||||
* <p/>Note: This only adds the edge information. Vertex
|
||||
* endpoint information needs to be written to the vertex
|
||||
* table via {@link VertexTableWrapper}.
|
||||
* @param edge
|
||||
*/
|
||||
public void writeEdge(Edge edge) {
|
||||
Mutators.apply(getWriter(), new EdgeMutator.Add(edge));
|
||||
globals.checkedFlush();
|
||||
}
|
||||
|
||||
public void deleteEdge(Edge edge) {
|
||||
Mutators.apply(getWriter(), new EdgeMutator.Delete(edge));
|
||||
globals.checkedFlush();
|
||||
}
|
||||
|
||||
public CloseableIterable<Edge> getEdges() {
|
||||
Scanner scan = getScanner();
|
||||
scan.fetchColumnFamily(new Text(Constants.LABEL));
|
||||
|
||||
if (globals.getConfig().getPreloadedProperties() != null) {
|
||||
for (String key : globals.getConfig().getPreloadedProperties()) {
|
||||
scan.fetchColumnFamily(new Text(key));
|
||||
}
|
||||
}
|
||||
|
||||
final EdgeParser parser = new EdgeParser(globals);
|
||||
|
||||
return new ScannerIterable<Edge>(scan) {
|
||||
@Override
|
||||
public Edge next(PeekingIterator<Entry<Key, Value>> iterator) {
|
||||
// TODO could also check local cache before creating a new instance?
|
||||
|
||||
String rowId = iterator.peek().getKey().getRow().toString();
|
||||
|
||||
List<Entry<Key, Value>> entries =
|
||||
new ArrayList<Entry<Key, Value>>();
|
||||
|
||||
// MDL 05 Jan 2014: Why is this equalsIgnoreCase??
|
||||
while (iterator.peek() != null && rowId.equalsIgnoreCase(iterator
|
||||
.peek().getKey().getRow().toString())) {
|
||||
entries.add(iterator.next());
|
||||
}
|
||||
|
||||
AccumuloEdge edge = parser.parse(rowId, entries);
|
||||
globals.getCaches().cache(edge, Edge.class);
|
||||
|
||||
return edge;
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
public Iterable<Edge> getEdges(String key, Object value) {
|
||||
AccumuloGraphUtils.nullCheckProperty(key, value);
|
||||
if (key.equalsIgnoreCase("label")) {
|
||||
key = Constants.LABEL;
|
||||
}
|
||||
|
||||
BatchScanner scan = getBatchScanner();
|
||||
scan.fetchColumnFamily(new Text(key));
|
||||
|
||||
byte[] val = AccumuloByteSerializer.serialize(value);
|
||||
if (val[0] != AccumuloByteSerializer.SERIALIZABLE) {
|
||||
IteratorSetting is = new IteratorSetting(10, "filter", RegExFilter.class);
|
||||
RegExFilter.setRegexs(is, null, null, null, Pattern.quote(new String(val)), false);
|
||||
scan.addScanIterator(is);
|
||||
|
||||
return new ScannerIterable<Edge>(scan) {
|
||||
|
||||
@Override
|
||||
public Edge next(PeekingIterator<Entry<Key,Value>> iterator) {
|
||||
|
||||
Key k = iterator.next().getKey();
|
||||
|
||||
if (k.getColumnFamily().toString().equals(Constants.LABEL)) {
|
||||
String[] vals = k.getColumnQualifier().toString().split(Constants.ID_DELIM);
|
||||
return new AccumuloEdge(globals, k.getRow().toString(),
|
||||
new AccumuloVertex(globals, vals[0]),
|
||||
new AccumuloVertex(globals, vals[1]), null);
|
||||
}
|
||||
return new AccumuloEdge(globals, k.getRow().toString());
|
||||
}
|
||||
};
|
||||
} else {
|
||||
// TODO
|
||||
throw new UnsupportedOperationException("Filtering on binary data not currently supported.");
|
||||
}
|
||||
}
|
||||
|
||||
public void loadEndpointsAndLabel(AccumuloEdge edge) {
|
||||
Scanner s = getScanner();
|
||||
|
||||
try {
|
||||
s.setRange(new Range(edge.getId().toString()));
|
||||
s.fetchColumnFamily(new Text(Constants.LABEL));
|
||||
Iterator<Entry<Key,Value>> iter = s.iterator();
|
||||
if (!iter.hasNext()) {
|
||||
dump();
|
||||
throw new AccumuloGraphException("Unable to find edge row: "+edge);
|
||||
}
|
||||
|
||||
Entry<Key, Value> entry = iter.next();
|
||||
|
||||
String cq = entry.getKey().getColumnQualifier().toString();
|
||||
String[] ids = cq.split(Constants.ID_DELIM);
|
||||
|
||||
String label = AccumuloByteSerializer.deserialize(entry.getValue().get());
|
||||
|
||||
edge.setVertices(new AccumuloVertex(globals, ids[0]),
|
||||
new AccumuloVertex(globals, ids[1]));
|
||||
edge.setLabel(label);
|
||||
|
||||
} finally {
|
||||
s.close();
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,226 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop.tables.core;
|
||||
|
||||
import java.util.HashSet;
|
||||
import java.util.Iterator;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import java.util.Map.Entry;
|
||||
|
||||
import org.apache.accumulo.core.client.BatchWriter;
|
||||
import org.apache.accumulo.core.client.IteratorSetting;
|
||||
import org.apache.accumulo.core.client.Scanner;
|
||||
import org.apache.accumulo.core.data.Key;
|
||||
import org.apache.accumulo.core.data.Range;
|
||||
import org.apache.accumulo.core.data.Value;
|
||||
import org.apache.accumulo.core.iterators.user.RegExFilter;
|
||||
import org.apache.hadoop.io.Text;
|
||||
|
||||
import com.tinkerpop.blueprints.Element;
|
||||
import com.tinkerpop.blueprints.util.StringFactory;
|
||||
|
||||
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
|
||||
import edu.jhuapl.tinkerpop.Constants;
|
||||
import edu.jhuapl.tinkerpop.GlobalInstances;
|
||||
import edu.jhuapl.tinkerpop.mutator.property.ClearPropertyMutator;
|
||||
import edu.jhuapl.tinkerpop.mutator.property.WritePropertyMutator;
|
||||
import edu.jhuapl.tinkerpop.mutator.Mutators;
|
||||
import edu.jhuapl.tinkerpop.parser.PropertyParser;
|
||||
import edu.jhuapl.tinkerpop.tables.BaseTableWrapper;
|
||||
|
||||
/**
|
||||
* Wrapper around tables with operations
|
||||
* common to {@link Element}s.
|
||||
*/
|
||||
public abstract class ElementTableWrapper extends BaseTableWrapper {
|
||||
|
||||
private BatchWriter writer;
|
||||
|
||||
public ElementTableWrapper(GlobalInstances globals, String tableName) {
|
||||
super(globals, tableName);
|
||||
writer = super.getWriter();
|
||||
}
|
||||
|
||||
/**
|
||||
* Give a single instance of the writer for this table.
|
||||
*/
|
||||
@Override
|
||||
protected BatchWriter getWriter() {
|
||||
return writer;
|
||||
}
|
||||
|
||||
/**
|
||||
* Read the given property from the backing table
|
||||
* for the given element id.
|
||||
* @param id
|
||||
* @param key
|
||||
* @return
|
||||
*/
|
||||
public <V> V readProperty(Element element, String key) {
|
||||
Scanner s = getScanner();
|
||||
|
||||
s.setRange(new Range(element.getId().toString()));
|
||||
|
||||
Text colf = StringFactory.LABEL.equals(key)
|
||||
? new Text(Constants.LABEL) : new Text(key);
|
||||
s.fetchColumnFamily(colf);
|
||||
|
||||
V value = null;
|
||||
|
||||
Iterator<Entry<Key, Value>> iter = s.iterator();
|
||||
if (iter.hasNext()) {
|
||||
value = AccumuloByteSerializer.deserialize(iter.next().getValue().get());
|
||||
}
|
||||
s.close();
|
||||
|
||||
return value;
|
||||
}
|
||||
|
||||
/**
|
||||
* Read all properties for the given element
|
||||
* from the backing table.
|
||||
* If the element has no properties, return an empty Map.
|
||||
* If the element does not exist, return null.
|
||||
* @param element
|
||||
* @return
|
||||
*/
|
||||
public Map<String, Object> readAllProperties(Element element) {
|
||||
return readProperties(element, null);
|
||||
}
|
||||
|
||||
/**
|
||||
* Read the given properties for the given element.
|
||||
* If propertyKeys is null, read all properties.
|
||||
* If the element has no properties, return an empty Map.
|
||||
* If the element does not exist, return null.
|
||||
* @param id
|
||||
* @param propertyKeys
|
||||
* @return
|
||||
*/
|
||||
public Map<String, Object> readProperties(Element element, String[] propertyKeys) {
|
||||
Scanner s = getScanner();
|
||||
s.setRange(Range.exact((String) element.getId()));
|
||||
|
||||
// If propertyKeys is null, we read everything.
|
||||
// Otherwise, limit to the given attributes.
|
||||
if (propertyKeys != null) {
|
||||
s.fetchColumnFamily(new Text(Constants.LABEL));
|
||||
|
||||
for (String key : propertyKeys) {
|
||||
s.fetchColumnFamily(new Text(key));
|
||||
}
|
||||
}
|
||||
|
||||
Map<String, Object> props = new PropertyParser().parse(s);
|
||||
s.close();
|
||||
|
||||
return props;
|
||||
}
|
||||
|
||||
/**
|
||||
* Return true if the element with given id exists.
|
||||
* @param id
|
||||
* @return
|
||||
*/
|
||||
public boolean elementExists(String id) {
|
||||
Scanner scan = null;
|
||||
try {
|
||||
scan = getScanner();
|
||||
scan.setRange(Range.exact(id));
|
||||
scan.fetchColumnFamily(new Text(Constants.LABEL));
|
||||
return new PropertyParser().parse(scan) != null;
|
||||
|
||||
} finally {
|
||||
if (scan != null) {
|
||||
scan.close();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Get all property keys for the given element id.
|
||||
* @param id
|
||||
* @return
|
||||
*/
|
||||
public Set<String> readPropertyKeys(Element element) {
|
||||
Scanner s = getScanner();
|
||||
|
||||
s.setRange(new Range(element.getId().toString()));
|
||||
|
||||
Set<String> keys = new HashSet<String>();
|
||||
|
||||
for (Entry<Key, Value> entry : s) {
|
||||
String cf = entry.getKey().getColumnFamily().toString();
|
||||
keys.add(cf);
|
||||
}
|
||||
|
||||
s.close();
|
||||
|
||||
// Remove some special keys.
|
||||
keys.remove(Constants.IN_EDGE);
|
||||
keys.remove(Constants.LABEL);
|
||||
keys.remove(Constants.OUT_EDGE);
|
||||
|
||||
return keys;
|
||||
}
|
||||
|
||||
/**
|
||||
* Delete the property entry from property table.
|
||||
* @param id
|
||||
* @param key
|
||||
*/
|
||||
public void clearProperty(Element element, String key) {
|
||||
Mutators.apply(getWriter(),
|
||||
new ClearPropertyMutator(element.getId().toString(), key));
|
||||
globals.checkedFlush();
|
||||
}
|
||||
|
||||
/**
|
||||
* Write the given property to the property table.
|
||||
* @param id
|
||||
* @param key
|
||||
* @param value
|
||||
*/
|
||||
public void writeProperty(Element element, String key, Object value) {
|
||||
Mutators.apply(getWriter(),
|
||||
new WritePropertyMutator(element.getId().toString(),
|
||||
key, value));
|
||||
globals.checkedFlush();
|
||||
}
|
||||
|
||||
/**
|
||||
* Add custom iterator to the given scanner so that
|
||||
* it will only return keys with value corresponding to an edge.
|
||||
* @param scan
|
||||
* @param labels
|
||||
*/
|
||||
protected void applyEdgeLabelValueFilter(Scanner scan, String... labels) {
|
||||
StringBuilder regex = new StringBuilder();
|
||||
for (String lab : labels) {
|
||||
if (regex.length() != 0)
|
||||
regex.append("|");
|
||||
regex.append(".*"+Constants.ID_DELIM+"\\Q").append(lab).append("\\E$");
|
||||
}
|
||||
|
||||
IteratorSetting is = new IteratorSetting(10, "edgeValueFilter", RegExFilter.class);
|
||||
RegExFilter.setRegexs(is, null, null, null, regex.toString(), false);
|
||||
scan.addScanIterator(is);
|
||||
}
|
||||
|
||||
public void close() {
|
||||
// TODO?
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,269 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop.tables.core;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.Arrays;
|
||||
import java.util.List;
|
||||
import java.util.Map.Entry;
|
||||
import java.util.regex.Pattern;
|
||||
|
||||
import org.apache.accumulo.core.client.BatchDeleter;
|
||||
import org.apache.accumulo.core.client.BatchScanner;
|
||||
import org.apache.accumulo.core.client.IteratorSetting;
|
||||
import org.apache.accumulo.core.client.Scanner;
|
||||
import org.apache.accumulo.core.data.Key;
|
||||
import org.apache.accumulo.core.data.Range;
|
||||
import org.apache.accumulo.core.data.Value;
|
||||
import org.apache.accumulo.core.iterators.user.RegExFilter;
|
||||
import org.apache.accumulo.core.util.PeekingIterator;
|
||||
import org.apache.hadoop.io.Text;
|
||||
|
||||
import com.tinkerpop.blueprints.CloseableIterable;
|
||||
import com.tinkerpop.blueprints.Direction;
|
||||
import com.tinkerpop.blueprints.Edge;
|
||||
import com.tinkerpop.blueprints.Vertex;
|
||||
|
||||
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
|
||||
import edu.jhuapl.tinkerpop.AccumuloEdge;
|
||||
import edu.jhuapl.tinkerpop.AccumuloElement;
|
||||
import edu.jhuapl.tinkerpop.AccumuloGraphException;
|
||||
import edu.jhuapl.tinkerpop.AccumuloGraphUtils;
|
||||
import edu.jhuapl.tinkerpop.AccumuloVertex;
|
||||
import edu.jhuapl.tinkerpop.Constants;
|
||||
import edu.jhuapl.tinkerpop.GlobalInstances;
|
||||
import edu.jhuapl.tinkerpop.ScannerIterable;
|
||||
import edu.jhuapl.tinkerpop.mutator.vertex.AddVertexMutator;
|
||||
import edu.jhuapl.tinkerpop.mutator.Mutator;
|
||||
import edu.jhuapl.tinkerpop.mutator.Mutators;
|
||||
import edu.jhuapl.tinkerpop.mutator.edge.EdgeEndpointsMutator;
|
||||
import edu.jhuapl.tinkerpop.parser.VertexParser;
|
||||
|
||||
|
||||
/**
|
||||
* Wrapper around {@link Vertex} tables.
|
||||
*/
|
||||
public class VertexTableWrapper extends ElementTableWrapper {
|
||||
|
||||
public VertexTableWrapper(GlobalInstances globals) {
|
||||
super(globals, globals.getConfig().getVertexTableName());
|
||||
}
|
||||
|
||||
/**
|
||||
* Write a vertex with the given id.
|
||||
* Note: This does not currently write the vertex's properties.
|
||||
* @param vertex
|
||||
*/
|
||||
public void writeVertex(Vertex vertex) {
|
||||
Mutators.apply(getWriter(), new AddVertexMutator(vertex.getId().toString()));
|
||||
globals.checkedFlush();
|
||||
}
|
||||
|
||||
/**
|
||||
* Remove the given vertex.
|
||||
* Note: This uses a BatchDeleter rather than {@link Mutator}
|
||||
* because it is more efficient.
|
||||
* @param vertex
|
||||
*/
|
||||
public void deleteVertex(Vertex vertex) {
|
||||
BatchDeleter deleter = null;
|
||||
|
||||
try {
|
||||
deleter = getDeleter();
|
||||
deleter.setRanges(Arrays.asList(Range.exact((String) vertex.getId())));
|
||||
deleter.delete();
|
||||
|
||||
} catch (Exception e) {
|
||||
throw new AccumuloGraphException(e);
|
||||
} finally {
|
||||
if (deleter != null) {
|
||||
deleter.close();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Write edge endpoint information to the vertex table.
|
||||
* @param edge
|
||||
*/
|
||||
public void writeEdgeEndpoints(Edge edge) {
|
||||
Mutators.apply(getWriter(), new EdgeEndpointsMutator.Add(edge));
|
||||
globals.checkedFlush();
|
||||
}
|
||||
|
||||
public void deleteEdgeEndpoints(Edge edge) {
|
||||
Mutators.apply(getWriter(), new EdgeEndpointsMutator.Delete(edge));
|
||||
globals.checkedFlush();
|
||||
}
|
||||
|
||||
public CloseableIterable<Edge> getEdges(Vertex vertex, Direction direction,
|
||||
String... labels) {
|
||||
Scanner scan = getScanner();
|
||||
scan.setRange(new Range(vertex.getId().toString()));
|
||||
if (direction.equals(Direction.IN)) {
|
||||
scan.fetchColumnFamily(new Text(Constants.IN_EDGE));
|
||||
} else if (direction.equals(Direction.OUT)) {
|
||||
scan.fetchColumnFamily(new Text(Constants.OUT_EDGE));
|
||||
} else {
|
||||
scan.fetchColumnFamily(new Text(Constants.IN_EDGE));
|
||||
scan.fetchColumnFamily(new Text(Constants.OUT_EDGE));
|
||||
}
|
||||
|
||||
if (labels.length > 0) {
|
||||
applyEdgeLabelValueFilter(scan, labels);
|
||||
}
|
||||
|
||||
return new ScannerIterable<Edge>(scan) {
|
||||
|
||||
@Override
|
||||
public Edge next(PeekingIterator<Entry<Key,Value>> iterator) {
|
||||
// TODO better use of information readily available...
|
||||
// TODO could also check local cache before creating a new
|
||||
// instance?
|
||||
|
||||
Entry<Key,Value> kv = iterator.next();
|
||||
|
||||
String[] parts = kv.getKey().getColumnQualifier().toString().split(Constants.ID_DELIM);
|
||||
String label = (new String(kv.getValue().get())).split(Constants.ID_DELIM)[1];
|
||||
|
||||
AccumuloEdge edge;
|
||||
if (kv.getKey().getColumnFamily().toString().equalsIgnoreCase(Constants.IN_EDGE)) {
|
||||
edge = new AccumuloEdge(globals, parts[1],
|
||||
new AccumuloVertex(globals, kv.getKey().getRow().toString()),
|
||||
new AccumuloVertex(globals, parts[0]), label);
|
||||
} else {
|
||||
edge = new AccumuloEdge(globals, parts[1],
|
||||
new AccumuloVertex(globals, parts[0]),
|
||||
new AccumuloVertex(globals, kv.getKey().getRow().toString()), label);
|
||||
}
|
||||
globals.getCaches().cache(edge, Edge.class);
|
||||
|
||||
return edge;
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
public Iterable<Vertex> getVertices(Vertex vertex, Direction direction, String... labels) {
|
||||
Scanner scan = getScanner();
|
||||
scan.setRange(new Range(vertex.getId().toString()));
|
||||
if (direction.equals(Direction.IN)) {
|
||||
scan.fetchColumnFamily(new Text(Constants.IN_EDGE));
|
||||
} else if (direction.equals(Direction.OUT)) {
|
||||
scan.fetchColumnFamily(new Text(Constants.OUT_EDGE));
|
||||
} else {
|
||||
scan.fetchColumnFamily(new Text(Constants.IN_EDGE));
|
||||
scan.fetchColumnFamily(new Text(Constants.OUT_EDGE));
|
||||
}
|
||||
|
||||
if (labels != null && labels.length > 0) {
|
||||
applyEdgeLabelValueFilter(scan, labels);
|
||||
}
|
||||
|
||||
return new ScannerIterable<Vertex>(scan) {
|
||||
|
||||
@Override
|
||||
public Vertex next(PeekingIterator<Entry<Key,Value>> iterator) {
|
||||
// TODO better use of information readily available...
|
||||
// TODO could also check local cache before creating a new
|
||||
// instance?
|
||||
String[] parts = iterator.next().getKey().getColumnQualifier()
|
||||
.toString().split(Constants.ID_DELIM);
|
||||
|
||||
AccumuloVertex vertex = new AccumuloVertex(globals, parts[0]);
|
||||
globals.getCaches().cache(vertex, Vertex.class);
|
||||
|
||||
return vertex;
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
public CloseableIterable<Vertex> getVertices() {
|
||||
return getVerticesInRange(null, null);
|
||||
}
|
||||
|
||||
public CloseableIterable<Vertex> getVerticesInRange(Object fromId, Object toId) {
|
||||
Scanner scan = getScanner();
|
||||
scan.setRange(new Range(fromId != null ? fromId.toString() : null,
|
||||
toId != null ? toId.toString() : null));
|
||||
scan.fetchColumnFamily(new Text(Constants.LABEL));
|
||||
|
||||
if (globals.getConfig().getPreloadedProperties() != null) {
|
||||
for (String key : globals.getConfig().getPreloadedProperties()) {
|
||||
scan.fetchColumnFamily(new Text(key));
|
||||
}
|
||||
}
|
||||
|
||||
final VertexParser parser = new VertexParser(globals);
|
||||
|
||||
return new ScannerIterable<Vertex>(scan) {
|
||||
@Override
|
||||
public Vertex next(PeekingIterator<Entry<Key, Value>> iterator) {
|
||||
// TODO could also check local cache before creating a new instance?
|
||||
|
||||
String rowId = iterator.peek().getKey().getRow().toString();
|
||||
|
||||
List<Entry<Key, Value>> entries =
|
||||
new ArrayList<Entry<Key, Value>>();
|
||||
|
||||
while (iterator.peek() != null && rowId.equals(iterator
|
||||
.peek().getKey().getRow().toString())) {
|
||||
entries.add(iterator.next());
|
||||
}
|
||||
|
||||
AccumuloVertex vertex = parser.parse(rowId, entries);
|
||||
globals.getCaches().cache(vertex, Vertex.class);
|
||||
|
||||
return vertex;
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
public Iterable<Vertex> getVertices(String key, Object value) {
|
||||
AccumuloGraphUtils.validateProperty(key, value);
|
||||
|
||||
byte[] val = AccumuloByteSerializer.serialize(value);
|
||||
if (val[0] != AccumuloByteSerializer.SERIALIZABLE) {
|
||||
BatchScanner scan = getBatchScanner();
|
||||
scan.fetchColumnFamily(new Text(key));
|
||||
|
||||
IteratorSetting is = new IteratorSetting(10, "filter", RegExFilter.class);
|
||||
RegExFilter.setRegexs(is, null, null, null, Pattern.quote(new String(val)), false);
|
||||
scan.addScanIterator(is);
|
||||
|
||||
return new ScannerIterable<Vertex>(scan) {
|
||||
|
||||
@Override
|
||||
public Vertex next(PeekingIterator<Entry<Key,Value>> iterator) {
|
||||
Entry<Key, Value> kv = iterator.next();
|
||||
String key = kv.getKey().getColumnFamily().toString();
|
||||
Object value = AccumuloByteSerializer.deserialize(kv.getValue().get());
|
||||
|
||||
Vertex v = globals.getCaches().retrieve(kv.getKey().getRow().toString(), Vertex.class);
|
||||
if (v == null) {
|
||||
v = new AccumuloVertex(globals, kv.getKey().getRow().toString());
|
||||
}
|
||||
|
||||
((AccumuloElement) v).setPropertyInMemory(key, value);
|
||||
globals.getCaches().cache(v, Vertex.class);
|
||||
|
||||
return v;
|
||||
}
|
||||
};
|
||||
} else {
|
||||
// TODO
|
||||
throw new UnsupportedOperationException("Filtering on binary data not currently supported.");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,179 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop.tables.index;
|
||||
|
||||
import java.util.Arrays;
|
||||
import java.util.Collections;
|
||||
import java.util.Map.Entry;
|
||||
import java.util.regex.Pattern;
|
||||
|
||||
import org.apache.accumulo.core.client.BatchDeleter;
|
||||
import org.apache.accumulo.core.client.BatchWriter;
|
||||
import org.apache.accumulo.core.client.IteratorSetting;
|
||||
import org.apache.accumulo.core.client.Scanner;
|
||||
import org.apache.accumulo.core.data.Key;
|
||||
import org.apache.accumulo.core.data.Range;
|
||||
import org.apache.accumulo.core.data.Value;
|
||||
import org.apache.accumulo.core.iterators.user.RegExFilter;
|
||||
import org.apache.accumulo.core.util.PeekingIterator;
|
||||
import org.apache.hadoop.io.Text;
|
||||
|
||||
import com.tinkerpop.blueprints.CloseableIterable;
|
||||
import com.tinkerpop.blueprints.Element;
|
||||
import com.tinkerpop.blueprints.IndexableGraph;
|
||||
import com.tinkerpop.blueprints.Vertex;
|
||||
|
||||
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
|
||||
import edu.jhuapl.tinkerpop.AccumuloElement;
|
||||
import edu.jhuapl.tinkerpop.AccumuloGraphException;
|
||||
import edu.jhuapl.tinkerpop.AccumuloGraphUtils;
|
||||
import edu.jhuapl.tinkerpop.GlobalInstances;
|
||||
import edu.jhuapl.tinkerpop.ScannerIterable;
|
||||
import edu.jhuapl.tinkerpop.mutator.Mutators;
|
||||
import edu.jhuapl.tinkerpop.mutator.index.IndexValueMutator;
|
||||
import edu.jhuapl.tinkerpop.parser.EdgeIndexParser;
|
||||
import edu.jhuapl.tinkerpop.parser.ElementIndexParser;
|
||||
import edu.jhuapl.tinkerpop.parser.VertexIndexParser;
|
||||
import edu.jhuapl.tinkerpop.tables.BaseTableWrapper;
|
||||
|
||||
/**
|
||||
* Wrapper around index tables containing properties
|
||||
* and values.
|
||||
*/
|
||||
public abstract class BaseIndexValuesTableWrapper extends BaseTableWrapper {
|
||||
|
||||
protected final Class<? extends Element> elementType;
|
||||
|
||||
protected BaseIndexValuesTableWrapper(GlobalInstances globals,
|
||||
Class<? extends Element> elementType, String tableName) {
|
||||
super(globals, tableName);
|
||||
this.elementType = elementType;
|
||||
}
|
||||
|
||||
/**
|
||||
* Return class of this index.
|
||||
* @return
|
||||
*/
|
||||
public Class<? extends Element> getElementType() {
|
||||
return elementType;
|
||||
}
|
||||
|
||||
/**
|
||||
* Add the property to this index, if autoindexing is enabled
|
||||
* and/or the given key has indexing enabled.
|
||||
* @param element
|
||||
* @param key
|
||||
* @param value
|
||||
*/
|
||||
public void setPropertyForIndex(Element element, String key, Object value) {
|
||||
setPropertyForIndex(element, key, value, false);
|
||||
}
|
||||
|
||||
/**
|
||||
* Add the property to this index.
|
||||
*
|
||||
* <p/>Note that this requires a round-trip to Accumulo to see
|
||||
* if the property exists if the provided key has an index.
|
||||
* So for best performance, create indices after bulk ingest.
|
||||
* <p/>If the force parameter is true, set the property regardless
|
||||
* of whether indexing is enabled for the given key. This is needed
|
||||
* for {@link IndexableGraph} operations.
|
||||
* @param element
|
||||
* @param key
|
||||
* @param value
|
||||
* @param force
|
||||
*/
|
||||
public void setPropertyForIndex(Element element, String key, Object value,
|
||||
boolean force) {
|
||||
AccumuloGraphUtils.validateProperty(key, value);
|
||||
if (force || globals.getConfig().getAutoIndex() ||
|
||||
globals.getIndexMetadataWrapper()
|
||||
.getIndexedKeys(elementType).contains(key)) {
|
||||
BatchWriter writer = getWriter();
|
||||
|
||||
Object oldValue = element.getProperty(key);
|
||||
if (oldValue != null && !oldValue.equals(value)) {
|
||||
Mutators.apply(writer, new IndexValueMutator.Delete(element, key, oldValue));
|
||||
}
|
||||
|
||||
Mutators.apply(writer, new IndexValueMutator.Add(element, key, value));
|
||||
globals.checkedFlush();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Remove property from the index.
|
||||
* @param element
|
||||
* @param key
|
||||
* @param value
|
||||
*/
|
||||
public void removePropertyFromIndex(Element element, String key, Object value) {
|
||||
if (value != null) {
|
||||
Mutators.apply(getWriter(), new IndexValueMutator.Delete(element, key, value));
|
||||
globals.checkedFlush();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Get elements with the key/value pair.
|
||||
* @param key
|
||||
* @param value
|
||||
* @return
|
||||
*/
|
||||
@SuppressWarnings("unchecked")
|
||||
public <T extends Element> CloseableIterable<T> readElementsFromIndex(String key, Object value) {
|
||||
Scanner scan = getScanner();
|
||||
byte[] id = AccumuloByteSerializer.serialize(value);
|
||||
scan.setRange(Range.exact(new Text(id)));
|
||||
scan.fetchColumnFamily(new Text(key));
|
||||
|
||||
final ElementIndexParser<? extends AccumuloElement> parser =
|
||||
Vertex.class.equals(elementType) ? new VertexIndexParser(globals) :
|
||||
new EdgeIndexParser(globals);
|
||||
|
||||
return new ScannerIterable<T>(scan) {
|
||||
@Override
|
||||
public T next(PeekingIterator<Entry<Key,Value>> iterator) {
|
||||
return (T) parser.parse(Arrays.asList(iterator.next()));
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Remove the given element's properties from the index.
|
||||
* @param element
|
||||
*/
|
||||
public void removeElementFromIndex(Element element) {
|
||||
BatchDeleter deleter = null;
|
||||
|
||||
try {
|
||||
deleter = getDeleter();
|
||||
deleter.setRanges(Collections.singleton(new Range()));
|
||||
|
||||
IteratorSetting is = new IteratorSetting(10, "getEdgeFilter", RegExFilter.class);
|
||||
RegExFilter.setRegexs(is, null, null,
|
||||
"^"+Pattern.quote(element.getId().toString())+"$", null, false);
|
||||
deleter.addScanIterator(is);
|
||||
deleter.delete();
|
||||
deleter.close();
|
||||
} catch (Exception e) {
|
||||
throw new AccumuloGraphException(e);
|
||||
} finally {
|
||||
if (deleter != null) {
|
||||
deleter.close();
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,77 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop.tables.index;
|
||||
|
||||
import com.tinkerpop.blueprints.CloseableIterable;
|
||||
import com.tinkerpop.blueprints.Edge;
|
||||
import com.tinkerpop.blueprints.Element;
|
||||
import com.tinkerpop.blueprints.Vertex;
|
||||
|
||||
import edu.jhuapl.tinkerpop.AccumuloGraphException;
|
||||
import edu.jhuapl.tinkerpop.GlobalInstances;
|
||||
import edu.jhuapl.tinkerpop.tables.core.EdgeTableWrapper;
|
||||
import edu.jhuapl.tinkerpop.tables.core.ElementTableWrapper;
|
||||
import edu.jhuapl.tinkerpop.tables.core.VertexTableWrapper;
|
||||
|
||||
/**
|
||||
* Base class for key index tables.
|
||||
*/
|
||||
public abstract class BaseKeyIndexTableWrapper extends BaseIndexValuesTableWrapper {
|
||||
|
||||
protected BaseKeyIndexTableWrapper(GlobalInstances globals,
|
||||
Class<? extends Element> elementType, String tableName) {
|
||||
super(globals, elementType, tableName);
|
||||
}
|
||||
|
||||
/**
|
||||
* Rebuild this index for the given table.
|
||||
* @param table
|
||||
* @param key
|
||||
*/
|
||||
public void rebuildIndex(String key, Class<? extends Element> elementClass) {
|
||||
ElementTableWrapper wrapper = globals.getElementWrapper(elementClass);
|
||||
if (wrapper instanceof VertexTableWrapper) {
|
||||
CloseableIterable<Vertex> iter = ((VertexTableWrapper) wrapper).getVertices();
|
||||
for (Vertex v : iter) {
|
||||
rebuild(wrapper, v, key);
|
||||
}
|
||||
iter.close();
|
||||
}
|
||||
else if (wrapper instanceof EdgeTableWrapper) {
|
||||
CloseableIterable<Edge> iter = ((EdgeTableWrapper) wrapper).getEdges();
|
||||
for (Edge e : iter) {
|
||||
rebuild(wrapper, e, key);
|
||||
}
|
||||
iter.close();
|
||||
}
|
||||
else {
|
||||
throw new AccumuloGraphException("Unexpected table wrapper: "+wrapper.getClass());
|
||||
}
|
||||
globals.checkedFlush();
|
||||
}
|
||||
|
||||
/**
|
||||
* Add given element to index for the given key.
|
||||
* @param element
|
||||
* @param key
|
||||
*/
|
||||
private void rebuild(ElementTableWrapper wrapper,
|
||||
Element element, String key) {
|
||||
Object value = wrapper.readProperty(element, key);
|
||||
if (value != null) {
|
||||
setPropertyForIndex(element, key, value);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,86 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop.tables.index;
|
||||
|
||||
import java.util.Arrays;
|
||||
import java.util.Map.Entry;
|
||||
|
||||
import org.apache.accumulo.core.client.Scanner;
|
||||
import org.apache.accumulo.core.data.Key;
|
||||
import org.apache.accumulo.core.data.Range;
|
||||
import org.apache.accumulo.core.data.Value;
|
||||
import org.apache.accumulo.core.util.PeekingIterator;
|
||||
import org.apache.hadoop.io.Text;
|
||||
|
||||
import com.tinkerpop.blueprints.Edge;
|
||||
|
||||
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
|
||||
import edu.jhuapl.tinkerpop.AccumuloEdge;
|
||||
import edu.jhuapl.tinkerpop.GlobalInstances;
|
||||
import edu.jhuapl.tinkerpop.ScannerIterable;
|
||||
import edu.jhuapl.tinkerpop.parser.EdgeIndexParser;
|
||||
|
||||
/**
|
||||
* Wrapper around {@link Edge} index table.
|
||||
*/
|
||||
public class EdgeKeyIndexTableWrapper extends BaseKeyIndexTableWrapper {
|
||||
|
||||
public EdgeKeyIndexTableWrapper(GlobalInstances globals) {
|
||||
super(globals, Edge.class, globals.getConfig()
|
||||
.getEdgeKeyIndexTableName());
|
||||
}
|
||||
|
||||
/**
|
||||
* Retrieve edges from the index table based
|
||||
* on the given key/value.
|
||||
* @param key
|
||||
* @param value
|
||||
* @return
|
||||
*/
|
||||
public Iterable<Edge> getEdges(String key, Object value) {
|
||||
Scanner s = getScanner();
|
||||
|
||||
Text row = new Text(AccumuloByteSerializer.serialize(value));
|
||||
s.setRange(Range.exact(row));
|
||||
s.fetchColumnFamily(new Text(key));
|
||||
|
||||
final EdgeIndexParser parser = new EdgeIndexParser(globals);
|
||||
|
||||
return new ScannerIterable<Edge>(s) {
|
||||
|
||||
@Override
|
||||
public Edge next(PeekingIterator<Entry<Key, Value>> iterator) {
|
||||
Entry<Key, Value> entry = iterator.next();
|
||||
AccumuloEdge e = parser.parse(Arrays.asList(entry));
|
||||
|
||||
// Check if we have it cached already, in which
|
||||
// case use the cached version.
|
||||
AccumuloEdge cached = (AccumuloEdge) globals.getCaches()
|
||||
.retrieve(e.getId(), Edge.class);
|
||||
if (cached != null) {
|
||||
for (String key : e.getPropertyKeysInMemory()) {
|
||||
cached.setPropertyInMemory(key, e.getPropertyInMemory(key));
|
||||
}
|
||||
|
||||
return cached;
|
||||
}
|
||||
|
||||
// We don't have it, so cache the new one and return it.
|
||||
globals.getCaches().cache(e, Edge.class);
|
||||
return e;
|
||||
}
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,170 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop.tables.index;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.HashSet;
|
||||
import java.util.List;
|
||||
import java.util.Set;
|
||||
|
||||
import org.apache.accumulo.core.client.Scanner;
|
||||
import org.apache.hadoop.io.Text;
|
||||
|
||||
import com.tinkerpop.blueprints.Element;
|
||||
import com.tinkerpop.blueprints.Index;
|
||||
import com.tinkerpop.blueprints.IndexableGraph;
|
||||
import com.tinkerpop.blueprints.KeyIndexableGraph;
|
||||
import com.tinkerpop.blueprints.util.ExceptionFactory;
|
||||
|
||||
import edu.jhuapl.tinkerpop.AccumuloIndex;
|
||||
import edu.jhuapl.tinkerpop.Constants.IndexMetadataEntryType;
|
||||
import edu.jhuapl.tinkerpop.GlobalInstances;
|
||||
import edu.jhuapl.tinkerpop.mutator.Mutators;
|
||||
import edu.jhuapl.tinkerpop.mutator.index.IndexMetadataMutator;
|
||||
import edu.jhuapl.tinkerpop.parser.IndexedItem;
|
||||
import edu.jhuapl.tinkerpop.parser.IndexedItemsListParser;
|
||||
import edu.jhuapl.tinkerpop.tables.BaseTableWrapper;
|
||||
|
||||
/**
|
||||
* Stores metadata, in particular the indexed keys
|
||||
* for {@link KeyIndexableGraph}, and the list of
|
||||
* named indexes for {@link IndexableGraph}.
|
||||
*/
|
||||
public class IndexMetadataTableWrapper extends BaseTableWrapper {
|
||||
|
||||
public IndexMetadataTableWrapper(GlobalInstances globals) {
|
||||
super(globals, globals.getConfig().getIndexMetadataTableName());
|
||||
}
|
||||
|
||||
|
||||
//////// Methods for KeyIndexableGraph ////////
|
||||
|
||||
public void writeKeyMetadataEntry(String key, Class<? extends Element> clazz) {
|
||||
Mutators.apply(getWriter(), new IndexMetadataMutator.Add(key, clazz,
|
||||
IndexMetadataEntryType.__INDEX_KEY__));
|
||||
}
|
||||
|
||||
public void clearKeyMetadataEntry(String key, Class<? extends Element> clazz) {
|
||||
Mutators.apply(getWriter(), new IndexMetadataMutator.Delete(key, clazz,
|
||||
IndexMetadataEntryType.__INDEX_KEY__));
|
||||
}
|
||||
|
||||
public <T extends Element> Set<String> getIndexedKeys(Class<T> elementClass) {
|
||||
if (elementClass == null) {
|
||||
throw ExceptionFactory.classForElementCannotBeNull();
|
||||
}
|
||||
|
||||
IndexedItemsListParser parser = new IndexedItemsListParser(elementClass);
|
||||
|
||||
Scanner scan = null;
|
||||
try {
|
||||
scan = getScanner();
|
||||
scan.fetchColumnFamily(new Text(IndexMetadataEntryType.__INDEX_KEY__.name()));
|
||||
|
||||
Set<String> keys = new HashSet<String>();
|
||||
for (IndexedItem item : parser.parse(scan)) {
|
||||
keys.add(item.getKey());
|
||||
}
|
||||
|
||||
return keys;
|
||||
|
||||
} finally {
|
||||
if (scan != null) {
|
||||
scan.close();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
//////// Methods for IndexableGraph ////////
|
||||
|
||||
@SuppressWarnings({"rawtypes", "unchecked"})
|
||||
public Iterable<Index<? extends Element>> getIndices() {
|
||||
List<Index<? extends Element>> indexes = new ArrayList<Index<? extends Element>>();
|
||||
|
||||
IndexedItemsListParser parser = new IndexedItemsListParser();
|
||||
|
||||
Scanner scan = null;
|
||||
try {
|
||||
scan = getScanner();
|
||||
scan.fetchColumnFamily(new Text(IndexMetadataEntryType.__INDEX_NAME__.name()));
|
||||
|
||||
for (IndexedItem item : parser.parse(scan)) {
|
||||
indexes.add(new AccumuloIndex(globals,
|
||||
item.getKey(), item.getElementClass()));
|
||||
}
|
||||
|
||||
return indexes;
|
||||
|
||||
} finally {
|
||||
if (scan != null) {
|
||||
scan.close();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public <T extends Element> Index<T> getIndex(String indexName,
|
||||
Class<T> indexClass) {
|
||||
IndexedItemsListParser parser = new IndexedItemsListParser();
|
||||
|
||||
Scanner scan = null;
|
||||
try {
|
||||
scan = getScanner();
|
||||
scan.fetchColumnFamily(new Text(IndexMetadataEntryType.__INDEX_NAME__.name()));
|
||||
|
||||
for (IndexedItem item : parser.parse(scan)) {
|
||||
if (item.getKey().equals(indexName)) {
|
||||
if (item.getElementClass().equals(indexClass)) {
|
||||
return new AccumuloIndex<T>(globals, indexName,
|
||||
indexClass);
|
||||
}
|
||||
else {
|
||||
throw ExceptionFactory.indexDoesNotSupportClass(indexName, indexClass);
|
||||
}
|
||||
}
|
||||
}
|
||||
return null;
|
||||
|
||||
} finally {
|
||||
if (scan != null) {
|
||||
scan.close();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public <T extends Element> Index<T> createIndex(String indexName,
|
||||
Class<T> indexClass) {
|
||||
for (Index<?> index : getIndices()) {
|
||||
if (index.getIndexName().equals(indexName)) {
|
||||
throw ExceptionFactory.indexAlreadyExists(indexName);
|
||||
}
|
||||
}
|
||||
|
||||
writeIndexNameEntry(indexName, indexClass);
|
||||
return new AccumuloIndex<T>(globals, indexName, indexClass);
|
||||
}
|
||||
|
||||
private void writeIndexNameEntry(String indexName,
|
||||
Class<? extends Element> indexClass) {
|
||||
Mutators.apply(getWriter(), new IndexMetadataMutator.Add(indexName,
|
||||
indexClass, IndexMetadataEntryType.__INDEX_NAME__));
|
||||
}
|
||||
|
||||
public void clearIndexNameEntry(String indexName,
|
||||
Class<? extends Element> indexClass) {
|
||||
Mutators.apply(getWriter(), new IndexMetadataMutator.Delete(indexName,
|
||||
indexClass, IndexMetadataEntryType.__INDEX_NAME__));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop.tables.index;
|
||||
|
||||
import com.tinkerpop.blueprints.Element;
|
||||
import com.tinkerpop.blueprints.IndexableGraph;
|
||||
|
||||
import edu.jhuapl.tinkerpop.GlobalInstances;
|
||||
|
||||
/**
|
||||
* Wrapper around a named index table (for {@link IndexableGraph}).
|
||||
*/
|
||||
public class NamedIndexTableWrapper extends BaseIndexValuesTableWrapper {
|
||||
|
||||
public NamedIndexTableWrapper(GlobalInstances globals,
|
||||
Class<? extends Element> elementType, String indexName) {
|
||||
super(globals, elementType,
|
||||
globals.getConfig().getNamedIndexTableName(indexName));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop.tables.index;
|
||||
|
||||
import java.util.Arrays;
|
||||
import java.util.Map.Entry;
|
||||
|
||||
import org.apache.accumulo.core.client.Scanner;
|
||||
import org.apache.accumulo.core.data.Key;
|
||||
import org.apache.accumulo.core.data.Range;
|
||||
import org.apache.accumulo.core.data.Value;
|
||||
import org.apache.accumulo.core.util.PeekingIterator;
|
||||
import org.apache.hadoop.io.Text;
|
||||
|
||||
import com.tinkerpop.blueprints.Vertex;
|
||||
|
||||
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
|
||||
import edu.jhuapl.tinkerpop.AccumuloVertex;
|
||||
import edu.jhuapl.tinkerpop.GlobalInstances;
|
||||
import edu.jhuapl.tinkerpop.ScannerIterable;
|
||||
import edu.jhuapl.tinkerpop.parser.VertexIndexParser;
|
||||
|
||||
/**
|
||||
* Wrapper around {@link Vertex} index table.
|
||||
*/
|
||||
public class VertexKeyIndexTableWrapper extends BaseKeyIndexTableWrapper {
|
||||
|
||||
public VertexKeyIndexTableWrapper(GlobalInstances globals) {
|
||||
super(globals, Vertex.class, globals.getConfig()
|
||||
.getVertexKeyIndexTableName());
|
||||
}
|
||||
|
||||
/**
|
||||
* Use the index to retrieve vertices with the
|
||||
* given key/value.
|
||||
* @param key
|
||||
* @param value
|
||||
*/
|
||||
public Iterable<Vertex> getVertices(String key, Object value) {
|
||||
Scanner s = getScanner();
|
||||
|
||||
Text row = new Text(AccumuloByteSerializer.serialize(value));
|
||||
s.setRange(Range.exact(row));
|
||||
s.fetchColumnFamily(new Text(key));
|
||||
|
||||
final VertexIndexParser parser = new VertexIndexParser(globals);
|
||||
|
||||
return new ScannerIterable<Vertex>(s) {
|
||||
|
||||
@Override
|
||||
public Vertex next(PeekingIterator<Entry<Key, Value>> iterator) {
|
||||
Entry<Key, Value> entry = iterator.next();
|
||||
AccumuloVertex v = parser.parse(Arrays.asList(entry));
|
||||
|
||||
// Check if we have it cached already, in which
|
||||
// case use the cached version.
|
||||
AccumuloVertex cached = (AccumuloVertex) globals.getCaches()
|
||||
.retrieve(v.getId(), Vertex.class);
|
||||
if (cached != null) {
|
||||
for (String key : v.getPropertyKeysInMemory()) {
|
||||
cached.setPropertyInMemory(key, v.getPropertyInMemory(key));
|
||||
}
|
||||
|
||||
return cached;
|
||||
}
|
||||
|
||||
// We don't have it, so cache the new one and return it.
|
||||
globals.getCaches().cache(v, Vertex.class);
|
||||
return v;
|
||||
}
|
||||
};
|
||||
}
|
||||
}
|
||||
6
src/main/resources/log4j.properties
Normal file
6
src/main/resources/log4j.properties
Normal file
@@ -0,0 +1,6 @@
|
||||
log4j.rootLogger=INFO, stdout
|
||||
|
||||
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
|
||||
log4j.appender.stdout.Target=System.out
|
||||
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
|
||||
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c - %m%n
|
||||
@@ -0,0 +1,64 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop;
|
||||
|
||||
import static org.junit.Assert.assertEquals;
|
||||
import static org.junit.Assert.assertNotNull;
|
||||
import static org.junit.Assert.assertTrue;
|
||||
|
||||
import java.util.Iterator;
|
||||
|
||||
import org.junit.Test;
|
||||
|
||||
import com.tinkerpop.blueprints.Direction;
|
||||
import com.tinkerpop.blueprints.Edge;
|
||||
import com.tinkerpop.blueprints.Vertex;
|
||||
|
||||
public class AccumuloBulkIngesterTest {
|
||||
|
||||
@Test
|
||||
public void testBulkIngester() throws Exception {
|
||||
AccumuloGraphConfiguration cfg = AccumuloGraphTestUtils.generateGraphConfig("propertyBuilder").setClear(true);
|
||||
|
||||
AccumuloBulkIngester ingester = new AccumuloBulkIngester(cfg);
|
||||
|
||||
for (String t : cfg.getTableNames()) {
|
||||
assertTrue(cfg.getConnector().tableOperations().exists(t));
|
||||
}
|
||||
|
||||
ingester.addVertex("A").finish();
|
||||
ingester.addVertex("B").add("P1", "V1").add("P2", "2").finish();
|
||||
ingester.addEdge("A", "B", "edge").add("P3", "V3").finish();
|
||||
ingester.shutdown(true);
|
||||
|
||||
AccumuloGraph graph = new AccumuloGraph(cfg.clone().setClear(false));
|
||||
Vertex v1 = graph.getVertex("A");
|
||||
assertNotNull(v1);
|
||||
|
||||
Iterator<Edge> it = v1.getEdges(Direction.OUT).iterator();
|
||||
assertTrue(it.hasNext());
|
||||
|
||||
Edge e = it.next();
|
||||
assertEquals("edge", e.getLabel());
|
||||
|
||||
Vertex v2 = e.getVertex(Direction.IN);
|
||||
assertEquals("B", v2.getId());
|
||||
assertEquals("V1", v2.getProperty("P1"));
|
||||
assertEquals("2", v2.getProperty("P2"));
|
||||
|
||||
graph.shutdown();
|
||||
}
|
||||
|
||||
}
|
||||
119
src/test/java/edu/jhuapl/tinkerpop/AccumuloElementTest.java
Normal file
119
src/test/java/edu/jhuapl/tinkerpop/AccumuloElementTest.java
Normal file
@@ -0,0 +1,119 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop;
|
||||
|
||||
import static org.junit.Assert.*;
|
||||
|
||||
import org.junit.Test;
|
||||
|
||||
import com.tinkerpop.blueprints.Edge;
|
||||
import com.tinkerpop.blueprints.Graph;
|
||||
import com.tinkerpop.blueprints.Vertex;
|
||||
|
||||
/**
|
||||
* Tests related to Accumulo elements.
|
||||
*/
|
||||
public class AccumuloElementTest {
|
||||
|
||||
@Test
|
||||
public void testNonStringIds() throws Exception {
|
||||
Graph graph = AccumuloGraphTestUtils.makeGraph("nonStringIds");
|
||||
|
||||
Object[] ids = new Object[] {
|
||||
10, 20, 30L, 40L,
|
||||
50.0f, 60.0f, 70.0d, 80.0d,
|
||||
(byte) 'a', (byte) 'b', 'c', 'd',
|
||||
"str1", "str2",
|
||||
new GenericObject("str3"), new GenericObject("str4"),
|
||||
};
|
||||
|
||||
Object[] edgeIds = new Object[] {
|
||||
100, 200, 300L, 400L,
|
||||
500.0f, 600.0f, 700.0d, 800.0d,
|
||||
(byte) 'e', (byte) 'f', 'g', 'h',
|
||||
"str5", "str6",
|
||||
new GenericObject("str7"), new GenericObject("str8"),
|
||||
};
|
||||
|
||||
for (int i = 0; i < ids.length; i++) {
|
||||
assertNull(graph.getVertex(ids[i]));
|
||||
Vertex v = graph.addVertex(ids[i]);
|
||||
assertNotNull(v);
|
||||
assertNotNull(graph.getVertex(ids[i]));
|
||||
}
|
||||
assertEquals(ids.length, count(graph.getVertices()));
|
||||
|
||||
for (int i = 1; i < edgeIds.length; i++) {
|
||||
assertNull(graph.getEdge(edgeIds[i-1]));
|
||||
Edge e = graph.addEdge(edgeIds[i-1],
|
||||
graph.getVertex(ids[i-1]),
|
||||
graph.getVertex(ids[i]), "label");
|
||||
assertNotNull(e);
|
||||
assertNotNull(graph.getEdge(edgeIds[i-1]));
|
||||
}
|
||||
assertEquals(edgeIds.length-1, count(graph.getEdges()));
|
||||
|
||||
graph.shutdown();
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testGetVerticesInRange() {
|
||||
AccumuloGraph graph = (AccumuloGraph) AccumuloGraphTestUtils
|
||||
.makeGraph("testGetVerticesInRange");
|
||||
|
||||
assertEquals(0, count(graph.getVerticesInRange(null, null)));
|
||||
for (int i = 0; i < 10; i++) {
|
||||
graph.addVertex(id(i));
|
||||
}
|
||||
|
||||
assertEquals(10, count(graph.getVerticesInRange(null, null)));
|
||||
assertEquals(0, count(graph.getVerticesInRange(id(20), null)));
|
||||
assertEquals(10, count(graph.getVerticesInRange(null, id(20))));
|
||||
assertEquals(0, count(graph.getVerticesInRange(id(20), id(20))));
|
||||
|
||||
for (int i = 0; i < 10; i++) {
|
||||
assertEquals(i+1, count(graph.getVerticesInRange(null, id(i))));
|
||||
assertEquals(10-i, count(graph.getVerticesInRange(id(i), null)));
|
||||
assertEquals(1, count(graph.getVerticesInRange(id(i), id(i))));
|
||||
}
|
||||
|
||||
graph.shutdown();
|
||||
}
|
||||
|
||||
private static String id(int idNum) {
|
||||
return String.format("%08d", idNum);
|
||||
}
|
||||
|
||||
private static int count(Iterable<?> iter) {
|
||||
int count = 0;
|
||||
for (@SuppressWarnings("unused") Object obj : iter) {
|
||||
count++;
|
||||
}
|
||||
return count;
|
||||
}
|
||||
|
||||
private static class GenericObject {
|
||||
private final Object id;
|
||||
|
||||
public GenericObject(Object id) {
|
||||
this.id = id;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString() {
|
||||
return "GenericObject [id=" + id + "]";
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,184 +1,309 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop;
|
||||
|
||||
import static org.junit.Assert.*;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collection;
|
||||
import java.util.Iterator;
|
||||
import java.util.List;
|
||||
|
||||
import javax.xml.namespace.QName;
|
||||
|
||||
import org.apache.hadoop.io.Text;
|
||||
import org.junit.Test;
|
||||
|
||||
import com.tinkerpop.blueprints.Direction;
|
||||
import com.tinkerpop.blueprints.Edge;
|
||||
import com.tinkerpop.blueprints.GraphFactory;
|
||||
import com.tinkerpop.blueprints.Vertex;
|
||||
|
||||
public class AccumuloGraphConfigurationTest {
|
||||
|
||||
@Test
|
||||
public void testSplits() throws Exception {
|
||||
AccumuloGraphConfiguration cfg;
|
||||
|
||||
// Tests for splits string.
|
||||
cfg = AccumuloGraphTestUtils.generateGraphConfig("nullSplits").setSplits((String) null);
|
||||
AccumuloGraph graph = (AccumuloGraph) GraphFactory.open(cfg.getConfiguration());
|
||||
for (String table : cfg.getTableNames()) {
|
||||
assertEquals(0, cfg.getConnector().tableOperations().listSplits(table).size());
|
||||
}
|
||||
graph.shutdown();
|
||||
|
||||
cfg = AccumuloGraphTestUtils.generateGraphConfig("emptySplits").setSplits("");
|
||||
graph = (AccumuloGraph) GraphFactory.open(cfg.getConfiguration());
|
||||
for (String table : cfg.getTableNames()) {
|
||||
assertEquals(0, cfg.getConnector().tableOperations().listSplits(table).size());
|
||||
}
|
||||
graph.shutdown();
|
||||
|
||||
cfg = AccumuloGraphTestUtils.generateGraphConfig("threeSplits").setSplits(" a b c ");
|
||||
graph = (AccumuloGraph) GraphFactory.open(cfg.getConfiguration());
|
||||
for (String table : cfg.getTableNames()) {
|
||||
Collection<Text> splits = cfg.getConnector().tableOperations().listSplits(table);
|
||||
assertEquals(3, splits.size());
|
||||
List<Text> arr = new ArrayList<Text>(splits);
|
||||
assertEquals("a", arr.get(0).toString());
|
||||
assertEquals("b", arr.get(1).toString());
|
||||
assertEquals("c", arr.get(2).toString());
|
||||
}
|
||||
graph.shutdown();
|
||||
|
||||
// Tests for splits array.
|
||||
cfg = AccumuloGraphTestUtils.generateGraphConfig("nullSplitsArray").setSplits((String[]) null);
|
||||
graph = (AccumuloGraph) GraphFactory.open(cfg.getConfiguration());
|
||||
for (String table : cfg.getTableNames()) {
|
||||
assertEquals(0, cfg.getConnector().tableOperations().listSplits(table).size());
|
||||
}
|
||||
graph.shutdown();
|
||||
|
||||
cfg = AccumuloGraphTestUtils.generateGraphConfig("emptySplitsArray").setSplits(new String[] {});
|
||||
graph = (AccumuloGraph) GraphFactory.open(cfg.getConfiguration());
|
||||
for (String table : cfg.getTableNames()) {
|
||||
assertEquals(0, cfg.getConnector().tableOperations().listSplits(table).size());
|
||||
}
|
||||
graph.shutdown();
|
||||
|
||||
cfg = AccumuloGraphTestUtils.generateGraphConfig("threeSplitsArray").setSplits(new String[] {"d", "e", "f"});
|
||||
graph = (AccumuloGraph) GraphFactory.open(cfg.getConfiguration());
|
||||
for (String table : cfg.getTableNames()) {
|
||||
Collection<Text> splits = cfg.getConnector().tableOperations().listSplits(table);
|
||||
assertEquals(3, splits.size());
|
||||
List<Text> arr = new ArrayList<Text>(splits);
|
||||
assertEquals("d", arr.get(0).toString());
|
||||
assertEquals("e", arr.get(1).toString());
|
||||
assertEquals("f", arr.get(2).toString());
|
||||
}
|
||||
graph.shutdown();
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testPropertyValues() throws Exception {
|
||||
AccumuloGraph graph = new AccumuloGraph(AccumuloGraphTestUtils.generateGraphConfig("propertyValues"));
|
||||
// Tests for serialization/deserialization of properties.
|
||||
QName qname = new QName("ns", "prop");
|
||||
Vertex v = graph.addVertex(null);
|
||||
v.setProperty("qname", qname);
|
||||
assertTrue(v.getProperty("qname") instanceof QName);
|
||||
assertTrue(qname.equals(v.getProperty("qname")));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testIsEmpty() throws Exception {
|
||||
AccumuloGraphConfiguration cfg = AccumuloGraphTestUtils.generateGraphConfig("isEmpty");
|
||||
AccumuloGraph graph = new AccumuloGraph(cfg);
|
||||
assertTrue(graph.isEmpty());
|
||||
|
||||
graph.addVertex("A");
|
||||
assertFalse(graph.isEmpty());
|
||||
|
||||
graph.clear();
|
||||
assertTrue(graph.isEmpty());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCreateAndClear() throws Exception {
|
||||
AccumuloGraphConfiguration cfg = AccumuloGraphTestUtils.generateGraphConfig("noCreate").setCreate(false);
|
||||
try {
|
||||
new AccumuloGraph(cfg);
|
||||
fail("Create is disabled and graph does not exist");
|
||||
} catch (Exception e) {
|
||||
assertTrue(true);
|
||||
}
|
||||
|
||||
cfg = AccumuloGraphTestUtils.generateGraphConfig("yesCreate").setCreate(true);
|
||||
for (String t : cfg.getTableNames()) {
|
||||
assertFalse(cfg.getConnector().tableOperations().exists(t));
|
||||
}
|
||||
AccumuloGraph graph = new AccumuloGraph(cfg);
|
||||
for (String t : cfg.getTableNames()) {
|
||||
assertTrue(cfg.getConnector().tableOperations().exists(t));
|
||||
}
|
||||
graph.shutdown();
|
||||
|
||||
graph = new AccumuloGraph(cfg.setCreate(false));
|
||||
assertTrue(graph.isEmpty());
|
||||
graph.addVertex("A");
|
||||
graph.addVertex("B");
|
||||
assertFalse(graph.isEmpty());
|
||||
graph.shutdown();
|
||||
|
||||
graph = new AccumuloGraph(cfg.setClear(true));
|
||||
assertTrue(graph.isEmpty());
|
||||
graph.shutdown();
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testBulkIngester() throws Exception {
|
||||
AccumuloGraphConfiguration cfg = AccumuloGraphTestUtils.generateGraphConfig("propertyBuilder").setClear(true);
|
||||
|
||||
AccumuloBulkIngester ingester = new AccumuloBulkIngester(cfg);
|
||||
|
||||
for (String t : cfg.getTableNames()) {
|
||||
assertTrue(cfg.getConnector().tableOperations().exists(t));
|
||||
}
|
||||
|
||||
ingester.addVertex("A").finish();
|
||||
ingester.addVertex("B").add("P1", "V1").add("P2", "2").finish();
|
||||
ingester.addEdge("A", "B", "edge").add("P3", "V3").finish();
|
||||
ingester.shutdown(true);
|
||||
|
||||
cfg.setClear(false);
|
||||
AccumuloGraph graph = new AccumuloGraph(cfg);
|
||||
Vertex v1 = graph.getVertex("A");
|
||||
assertNotNull(v1);
|
||||
|
||||
Iterator<Edge> it = v1.getEdges(Direction.OUT).iterator();
|
||||
assertTrue(it.hasNext());
|
||||
|
||||
Edge e = it.next();
|
||||
assertEquals("edge", e.getLabel());
|
||||
|
||||
Vertex v2 = e.getVertex(Direction.IN);
|
||||
assertEquals("B", v2.getId());
|
||||
assertEquals("V1", v2.getProperty("P1"));
|
||||
assertEquals("2", v2.getProperty("P2"));
|
||||
|
||||
graph.shutdown();
|
||||
}
|
||||
}
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop;
|
||||
|
||||
import static org.junit.Assert.assertEquals;
|
||||
import static org.junit.Assert.assertFalse;
|
||||
import static org.junit.Assert.assertNotNull;
|
||||
import static org.junit.Assert.assertTrue;
|
||||
import static org.junit.Assert.fail;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collection;
|
||||
import java.util.List;
|
||||
|
||||
import javax.xml.namespace.QName;
|
||||
|
||||
import org.apache.accumulo.core.client.Instance;
|
||||
import org.apache.accumulo.core.client.mock.MockInstance;
|
||||
import org.apache.commons.configuration.Configuration;
|
||||
import org.apache.hadoop.io.Text;
|
||||
import org.junit.Test;
|
||||
|
||||
import com.tinkerpop.blueprints.GraphFactory;
|
||||
import com.tinkerpop.blueprints.Vertex;
|
||||
|
||||
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration.InstanceType;
|
||||
|
||||
public class AccumuloGraphConfigurationTest {
|
||||
|
||||
@Test
|
||||
public void testConfigurationInterface() throws Exception {
|
||||
Configuration conf = AccumuloGraphTestUtils.generateGraphConfig("setPropsValid");
|
||||
for (String key : AccumuloGraphConfiguration.getValidInternalKeys()) {
|
||||
// This is bad... but we should allow them if they are valid keys.
|
||||
conf.setProperty(key, "value");
|
||||
}
|
||||
|
||||
conf = AccumuloGraphTestUtils.generateGraphConfig("setPropsInvalid");
|
||||
try {
|
||||
conf.setProperty("invalidKey", "value");
|
||||
fail();
|
||||
} catch (Exception e) { }
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testSplits() throws Exception {
|
||||
AccumuloGraphConfiguration cfg;
|
||||
|
||||
// Tests for splits string.
|
||||
cfg = AccumuloGraphTestUtils.generateGraphConfig("nullSplits").setSplits((String) null);
|
||||
AccumuloGraph graph = (AccumuloGraph) GraphFactory.open(cfg.getConfiguration());
|
||||
for (String table : cfg.getTableNames()) {
|
||||
assertEquals(0, cfg.getConnector().tableOperations().listSplits(table).size());
|
||||
}
|
||||
graph.shutdown();
|
||||
|
||||
cfg = AccumuloGraphTestUtils.generateGraphConfig("emptySplits").setSplits("");
|
||||
graph = (AccumuloGraph) GraphFactory.open(cfg.getConfiguration());
|
||||
for (String table : cfg.getTableNames()) {
|
||||
assertEquals(0, cfg.getConnector().tableOperations().listSplits(table).size());
|
||||
}
|
||||
graph.shutdown();
|
||||
|
||||
cfg = AccumuloGraphTestUtils.generateGraphConfig("threeSplits").setSplits(" a b c ");
|
||||
graph = (AccumuloGraph) GraphFactory.open(cfg.getConfiguration());
|
||||
for (String table : cfg.getTableNames()) {
|
||||
Collection<Text> splits = cfg.getConnector().tableOperations().listSplits(table);
|
||||
assertEquals(3, splits.size());
|
||||
List<Text> arr = new ArrayList<Text>(splits);
|
||||
assertEquals("a", arr.get(0).toString());
|
||||
assertEquals("b", arr.get(1).toString());
|
||||
assertEquals("c", arr.get(2).toString());
|
||||
}
|
||||
graph.shutdown();
|
||||
|
||||
// Tests for splits array.
|
||||
cfg = AccumuloGraphTestUtils.generateGraphConfig("nullSplitsArray").setSplits((String[]) null);
|
||||
graph = (AccumuloGraph) GraphFactory.open(cfg.getConfiguration());
|
||||
for (String table : cfg.getTableNames()) {
|
||||
assertEquals(0, cfg.getConnector().tableOperations().listSplits(table).size());
|
||||
}
|
||||
graph.shutdown();
|
||||
|
||||
cfg = AccumuloGraphTestUtils.generateGraphConfig("emptySplitsArray").setSplits(new String[] {});
|
||||
graph = (AccumuloGraph) GraphFactory.open(cfg.getConfiguration());
|
||||
for (String table : cfg.getTableNames()) {
|
||||
assertEquals(0, cfg.getConnector().tableOperations().listSplits(table).size());
|
||||
}
|
||||
graph.shutdown();
|
||||
|
||||
cfg = AccumuloGraphTestUtils.generateGraphConfig("threeSplitsArray").setSplits(new String[] {"d", "e", "f"});
|
||||
graph = (AccumuloGraph) GraphFactory.open(cfg.getConfiguration());
|
||||
for (String table : cfg.getTableNames()) {
|
||||
Collection<Text> splits = cfg.getConnector().tableOperations().listSplits(table);
|
||||
assertEquals(3, splits.size());
|
||||
List<Text> arr = new ArrayList<Text>(splits);
|
||||
assertEquals("d", arr.get(0).toString());
|
||||
assertEquals("e", arr.get(1).toString());
|
||||
assertEquals("f", arr.get(2).toString());
|
||||
}
|
||||
graph.shutdown();
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testPropertyValues() throws Exception {
|
||||
AccumuloGraph graph = new AccumuloGraph(AccumuloGraphTestUtils.generateGraphConfig("propertyValues"));
|
||||
// Tests for serialization/deserialization of properties.
|
||||
QName qname = new QName("ns", "prop");
|
||||
Vertex v = graph.addVertex(null);
|
||||
v.setProperty("qname", qname);
|
||||
assertTrue(v.getProperty("qname") instanceof QName);
|
||||
assertTrue(qname.equals(v.getProperty("qname")));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testIsEmpty() throws Exception {
|
||||
AccumuloGraphConfiguration cfg = AccumuloGraphTestUtils.generateGraphConfig("isEmpty");
|
||||
AccumuloGraph graph = new AccumuloGraph(cfg);
|
||||
assertTrue(graph.isEmpty());
|
||||
|
||||
graph.addVertex("A");
|
||||
assertFalse(graph.isEmpty());
|
||||
|
||||
graph.clear();
|
||||
assertTrue(graph.isEmpty());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCreateAndClear() throws Exception {
|
||||
AccumuloGraphConfiguration cfg = AccumuloGraphTestUtils.generateGraphConfig("noCreate").setCreate(false);
|
||||
try {
|
||||
new AccumuloGraph(cfg);
|
||||
fail("Create is disabled and graph does not exist");
|
||||
} catch (Exception e) {
|
||||
assertTrue(true);
|
||||
}
|
||||
|
||||
cfg = AccumuloGraphTestUtils.generateGraphConfig("yesCreate").setCreate(true);
|
||||
for (String t : cfg.getTableNames()) {
|
||||
assertFalse(cfg.getConnector().tableOperations().exists(t));
|
||||
}
|
||||
AccumuloGraph graph = new AccumuloGraph(cfg);
|
||||
for (String t : cfg.getTableNames()) {
|
||||
assertTrue(cfg.getConnector().tableOperations().exists(t));
|
||||
}
|
||||
graph.shutdown();
|
||||
|
||||
graph = new AccumuloGraph(cfg.clone().setCreate(false));
|
||||
assertTrue(graph.isEmpty());
|
||||
graph.addVertex("A");
|
||||
graph.addVertex("B");
|
||||
assertFalse(graph.isEmpty());
|
||||
graph.shutdown();
|
||||
|
||||
graph = new AccumuloGraph(cfg.clone().setClear(true));
|
||||
assertTrue(graph.isEmpty());
|
||||
graph.shutdown();
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testPrint() throws Exception {
|
||||
AccumuloGraphConfiguration cfg =
|
||||
AccumuloGraphTestUtils.generateGraphConfig("printTest");
|
||||
cfg.print();
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testInvalidCacheParams() throws Exception {
|
||||
int size = 100;
|
||||
int timeout = 30000;
|
||||
|
||||
AccumuloGraphConfiguration cfg = AccumuloGraphTestUtils
|
||||
.generateGraphConfig("cacheParams");
|
||||
cfg.validate();
|
||||
|
||||
|
||||
// Vertex cache.
|
||||
|
||||
assertFalse(cfg.getVertexCacheEnabled());
|
||||
|
||||
try {
|
||||
cfg.setVertexCacheParams(-1, timeout);
|
||||
fail();
|
||||
} catch (Exception e) { }
|
||||
|
||||
try {
|
||||
cfg.setVertexCacheParams(size, -1);
|
||||
fail();
|
||||
} catch (Exception e) { }
|
||||
|
||||
assertFalse(cfg.getVertexCacheEnabled());
|
||||
|
||||
cfg.setVertexCacheParams(size, timeout);
|
||||
cfg.validate();
|
||||
assertTrue(cfg.getVertexCacheEnabled());
|
||||
assertEquals(size, cfg.getVertexCacheSize());
|
||||
assertEquals(timeout, cfg.getVertexCacheTimeout());
|
||||
|
||||
cfg.setVertexCacheParams(-1, -1);
|
||||
cfg.validate();
|
||||
assertFalse(cfg.getVertexCacheEnabled());
|
||||
|
||||
|
||||
// Edge cache.
|
||||
|
||||
assertFalse(cfg.getEdgeCacheEnabled());
|
||||
|
||||
try {
|
||||
cfg.setEdgeCacheParams(-1, timeout);
|
||||
fail();
|
||||
} catch (Exception e) { }
|
||||
|
||||
try {
|
||||
cfg.setEdgeCacheParams(size, -1);
|
||||
fail();
|
||||
} catch (Exception e) { }
|
||||
|
||||
assertFalse(cfg.getEdgeCacheEnabled());
|
||||
|
||||
cfg.setEdgeCacheParams(size, timeout);
|
||||
cfg.validate();
|
||||
assertTrue(cfg.getEdgeCacheEnabled());
|
||||
assertEquals(size, cfg.getEdgeCacheSize());
|
||||
assertEquals(timeout, cfg.getEdgeCacheTimeout());
|
||||
|
||||
cfg.setEdgeCacheParams(-1, -1);
|
||||
cfg.validate();
|
||||
assertFalse(cfg.getEdgeCacheEnabled());
|
||||
}
|
||||
|
||||
/**
|
||||
* Test different kinds of graph names (hyphens, punctuation, etc).
|
||||
* @throws Exception
|
||||
*/
|
||||
@Test
|
||||
public void testGraphNames() throws Exception {
|
||||
AccumuloGraphConfiguration conf = new AccumuloGraphConfiguration();
|
||||
|
||||
String[] valid = new String[] {
|
||||
"alpha", "12345", "alnum12345",
|
||||
"12345alnum", "under_score1", "_under_score_2"};
|
||||
String[] invalid = new String[] {"hyph-en",
|
||||
"dot..s", "quo\"tes"};
|
||||
|
||||
for (String name : valid) {
|
||||
conf.setGraphName(name);
|
||||
}
|
||||
|
||||
for (String name : invalid) {
|
||||
try {
|
||||
conf.setGraphName(name);
|
||||
fail();
|
||||
} catch (Exception e) { }
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testPreloadedProperties() {
|
||||
// Don't allow "all" and "some" preloaded properties.
|
||||
AccumuloGraphConfiguration conf = new AccumuloGraphConfiguration();
|
||||
conf.setPreloadAllProperties(true);
|
||||
conf.setPreloadedProperties(new String[]{"one", "two", "three"});
|
||||
try {
|
||||
conf.validate();
|
||||
fail();
|
||||
} catch (Exception e) { }
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testImmutableConnector() throws Exception {
|
||||
AccumuloGraphConfiguration cfg = new AccumuloGraphConfiguration().setInstanceType(
|
||||
InstanceType.Mock).setGraphName("immutableConnector")
|
||||
.setCreate(true).setAutoFlush(false);
|
||||
|
||||
cfg.getConnector();
|
||||
|
||||
try {
|
||||
cfg.setCreate(false);
|
||||
fail();
|
||||
} catch (Exception e) { }
|
||||
|
||||
try {
|
||||
cfg.setAutoFlush(true);
|
||||
fail();
|
||||
} catch (Exception e) { }
|
||||
|
||||
assertTrue(cfg.getCreate());
|
||||
assertFalse(cfg.getAutoFlush());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testMockInstanceValue(){
|
||||
AccumuloGraphConfiguration conf = new AccumuloGraphConfiguration().setInstanceType(InstanceType.Mock);
|
||||
assertNotNull(conf.getInstanceName());
|
||||
assertEquals("mock-instance", conf.getInstanceName());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -88,6 +88,7 @@ public class AccumuloGraphTest extends GraphTest {
|
||||
printTestPerformance("GraphSONReaderTestSuite", this.stopWatch());
|
||||
}
|
||||
|
||||
@Override
|
||||
public void doTestSuite(final TestSuite testSuite) throws Exception {
|
||||
String doTest = System.getProperty("testTinkerGraph");
|
||||
if (doTest == null || doTest.equals("true")) {
|
||||
@@ -112,12 +113,21 @@ public class AccumuloGraphTest extends GraphTest {
|
||||
new AccumuloGraphTest().generateGraph();
|
||||
}
|
||||
|
||||
@Override
|
||||
public void dropGraph(final String graphDirectoryName) {
|
||||
if (graphDirectoryName != null) {
|
||||
((AccumuloGraph) generateGraph(graphDirectoryName)).clear();
|
||||
AccumuloGraphConfiguration cfg = AccumuloGraphTestUtils.generateGraphConfig(graphDirectoryName);
|
||||
try {
|
||||
for (String table : cfg.getConnector().tableOperations().list()) {
|
||||
cfg.getConnector().tableOperations().delete(table);
|
||||
}
|
||||
} catch (Exception e) {
|
||||
throw new AccumuloGraphException(e);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public Object convertId(final Object id) {
|
||||
return id.toString();
|
||||
}
|
||||
|
||||
@@ -14,15 +14,19 @@
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop;
|
||||
|
||||
import com.tinkerpop.blueprints.Graph;
|
||||
import com.tinkerpop.blueprints.GraphFactory;
|
||||
|
||||
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration.InstanceType;
|
||||
|
||||
public class AccumuloGraphTestUtils {
|
||||
|
||||
public static AccumuloGraphConfiguration generateGraphConfig(String graphDirectoryName) {
|
||||
AccumuloGraphConfiguration cfg = new AccumuloGraphConfiguration();
|
||||
cfg.setInstanceName("instanceName").setZookeeperHosts("ZookeeperHostsString");
|
||||
cfg.setUser("root").setPassword("".getBytes());
|
||||
cfg.setGraphName(graphDirectoryName).setCreate(true).autoFlush(true).setInstanceType(InstanceType.Mock);
|
||||
return cfg;
|
||||
return new AccumuloGraphConfiguration().setInstanceType(InstanceType.Mock)
|
||||
.setGraphName(graphDirectoryName).setCreate(true);
|
||||
}
|
||||
|
||||
public static Graph makeGraph(String name) {
|
||||
return GraphFactory.open(generateGraphConfig(name));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2,55 +2,117 @@ package edu.jhuapl.tinkerpop;
|
||||
|
||||
import static org.junit.Assert.*;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.Map.Entry;
|
||||
|
||||
import org.apache.accumulo.core.client.AccumuloException;
|
||||
import org.apache.accumulo.core.client.AccumuloSecurityException;
|
||||
import org.apache.accumulo.core.client.Scanner;
|
||||
import org.apache.accumulo.core.data.Key;
|
||||
import org.apache.accumulo.core.data.Value;
|
||||
import org.junit.Test;
|
||||
|
||||
import com.tinkerpop.blueprints.Edge;
|
||||
import com.tinkerpop.blueprints.Element;
|
||||
import com.tinkerpop.blueprints.GraphFactory;
|
||||
import com.tinkerpop.blueprints.Vertex;
|
||||
|
||||
public class AutoIndexTest {
|
||||
|
||||
@Test
|
||||
public void testIndexCreation() throws AccumuloException, AccumuloSecurityException, IOException, InterruptedException {
|
||||
AccumuloGraph ag = (AccumuloGraph) GraphFactory.open(AccumuloGraphTestUtils.generateGraphConfig("AutoIndexTest").setAutoIndex(true).getConfiguration());
|
||||
String VERT = "1234";
|
||||
String KEY = "name";
|
||||
String VALUE = "bananaman";
|
||||
public void testVertexAutoIndex() throws Exception {
|
||||
AccumuloGraph graph = (AccumuloGraph) GraphFactory.open(AccumuloGraphTestUtils
|
||||
.generateGraphConfig("VertexAutoIndexTest").setAutoIndex(true).getConfiguration());
|
||||
String id = "1234";
|
||||
String key = "name";
|
||||
String value = "bananaman";
|
||||
|
||||
Vertex v1 = ag.addVertex(VERT);
|
||||
v1.setProperty(KEY, VALUE);
|
||||
Vertex v1 = graph.addVertex(id);
|
||||
v1.setProperty(key, value);
|
||||
|
||||
Scanner scan = ag.getVertexIndexScanner();
|
||||
for (Entry<Key,Value> kv : scan) {
|
||||
assertEquals(new String(AccumuloByteSerializer.serialize(VALUE)), kv.getKey().getRow().toString());
|
||||
assertEquals(KEY, kv.getKey().getColumnFamily().toString());
|
||||
assertEquals(VERT, kv.getKey().getColumnQualifier().toString());
|
||||
Iterable<Element> elements = graph.getGlobals()
|
||||
.getVertexKeyIndexWrapper().readElementsFromIndex(key, value);
|
||||
int count = 0;
|
||||
for (Element element : elements) {
|
||||
assertTrue(element instanceof Vertex);
|
||||
assertEquals(id, element.getId());
|
||||
assertEquals(value, element.getProperty(key));
|
||||
count++;
|
||||
}
|
||||
assertEquals(1, count);
|
||||
|
||||
graph.removeVertex(v1);
|
||||
elements = graph.getGlobals()
|
||||
.getVertexKeyIndexWrapper().readElementsFromIndex(key, value);
|
||||
assertEquals(0, count(elements));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testRegularCreation() throws AccumuloException, AccumuloSecurityException, IOException, InterruptedException {
|
||||
AccumuloGraph ag = (AccumuloGraph) GraphFactory.open(AccumuloGraphTestUtils.generateGraphConfig("NoAutoIndexTest").getConfiguration());
|
||||
String VERT = "1234";
|
||||
String KEY = "name";
|
||||
String VALUE = "bananaman";
|
||||
public void testVertexNoAutoIndex() throws Exception {
|
||||
AccumuloGraph graph = (AccumuloGraph) GraphFactory.open(AccumuloGraphTestUtils
|
||||
.generateGraphConfig("VertexNoAutoIndexTest").getConfiguration());
|
||||
String id = "1234";
|
||||
String key = "name";
|
||||
String value = "bananaman";
|
||||
|
||||
Vertex v1 = ag.addVertex(VERT);
|
||||
v1.setProperty(KEY, VALUE);
|
||||
|
||||
Scanner scan = ag.getVertexIndexScanner();
|
||||
for (Entry<Key,Value> kv : scan) {
|
||||
assertTrue(false);
|
||||
}
|
||||
Vertex v1 = graph.addVertex(id);
|
||||
v1.setProperty(key, value);
|
||||
|
||||
Iterable<Element> elements = graph.getGlobals()
|
||||
.getVertexKeyIndexWrapper().readElementsFromIndex(key, value);
|
||||
assertEquals(0, count(elements));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testEdgeAutoIndex() throws Exception {
|
||||
AccumuloGraph graph = (AccumuloGraph) GraphFactory.open(AccumuloGraphTestUtils
|
||||
.generateGraphConfig("EdgeAutoIndex").setAutoIndex(true).getConfiguration());
|
||||
String id1 = "A";
|
||||
String id2 = "B";
|
||||
String eid = "X";
|
||||
String key = "name";
|
||||
String value = "bananaman";
|
||||
|
||||
Vertex v1 = graph.addVertex(id1);
|
||||
Vertex v2 = graph.addVertex(id2);
|
||||
Edge e = graph.addEdge(eid, v1, v2, "edge");
|
||||
e.setProperty(key, value);
|
||||
|
||||
Iterable<Element> elements = graph.getGlobals()
|
||||
.getEdgeKeyIndexWrapper().readElementsFromIndex(key, value);
|
||||
int count = 0;
|
||||
for (Element element : elements) {
|
||||
assertTrue(element instanceof Edge);
|
||||
assertEquals(eid, element.getId());
|
||||
assertEquals(value, element.getProperty(key));
|
||||
count++;
|
||||
}
|
||||
assertEquals(1, count);
|
||||
|
||||
graph.removeVertex(v1);
|
||||
elements = graph.getGlobals()
|
||||
.getEdgeKeyIndexWrapper().readElementsFromIndex(key, value);
|
||||
assertEquals(0, count(elements));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testEdgeNoAutoIndex() throws Exception {
|
||||
AccumuloGraph graph = (AccumuloGraph) GraphFactory.open(AccumuloGraphTestUtils
|
||||
.generateGraphConfig("EdgeNoAutoIndexTest").getConfiguration());
|
||||
String id1 = "A";
|
||||
String id2 = "B";
|
||||
String eid = "X";
|
||||
String key = "name";
|
||||
String value = "bananaman";
|
||||
|
||||
Vertex v1 = graph.addVertex(id1);
|
||||
Vertex v2 = graph.addVertex(id2);
|
||||
Edge e = graph.addEdge(eid, v1, v2, "edge");
|
||||
e.setProperty(key, value);
|
||||
|
||||
Iterable<Element> elements = graph.getGlobals()
|
||||
.getEdgeKeyIndexWrapper().readElementsFromIndex(key, value);
|
||||
assertEquals(0, count(elements));
|
||||
}
|
||||
|
||||
@SuppressWarnings("unused")
|
||||
private static int count(Iterable<?> it) {
|
||||
int count = 0;
|
||||
for (Object obj : it) {
|
||||
count++;
|
||||
}
|
||||
return count;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
package edu.jhuapl.tinkerpop;
|
||||
|
||||
import org.apache.commons.configuration.Configuration;
|
||||
import org.junit.Ignore;
|
||||
|
||||
import com.tinkerpop.blueprints.Graph;
|
||||
import com.tinkerpop.blueprints.GraphFactory;
|
||||
|
||||
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration.InstanceType;
|
||||
|
||||
/**
|
||||
* Run the test suite for a Distributed instance type.
|
||||
* <p/>Note: This is disabled by default since we can't
|
||||
* guarantee an Accumulo cluster setup in a test environment.
|
||||
* To use this, set the constants below and remove
|
||||
* the @Ignore annotation.
|
||||
*/
|
||||
@Ignore
|
||||
public class DistributedInstanceTest extends AccumuloGraphTest {
|
||||
|
||||
// Connection constants.
|
||||
private static final String ZOOKEEPERS = "zkhost-1,zkhost-2";
|
||||
private static final String INSTANCE = "accumulo-instance";
|
||||
private static final String USER = "user";
|
||||
private static final String PASSWORD = "password";
|
||||
|
||||
@Override
|
||||
public Graph generateGraph(String graphDirectoryName) {
|
||||
Configuration cfg = new AccumuloGraphConfiguration()
|
||||
.setInstanceType(InstanceType.Distributed)
|
||||
.setZooKeeperHosts(ZOOKEEPERS)
|
||||
.setInstanceName(INSTANCE)
|
||||
.setUser(USER).setPassword(PASSWORD)
|
||||
.setGraphName(graphDirectoryName)
|
||||
.setCreate(true);
|
||||
testGraphName.set(graphDirectoryName);
|
||||
return GraphFactory.open(cfg);
|
||||
}
|
||||
}
|
||||
108
src/test/java/edu/jhuapl/tinkerpop/ElementCacheTest.java
Normal file
108
src/test/java/edu/jhuapl/tinkerpop/ElementCacheTest.java
Normal file
@@ -0,0 +1,108 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop;
|
||||
|
||||
import org.junit.Test;
|
||||
|
||||
import static org.junit.Assert.*;
|
||||
|
||||
import com.tinkerpop.blueprints.Edge;
|
||||
import com.tinkerpop.blueprints.Element;
|
||||
import com.tinkerpop.blueprints.Graph;
|
||||
import com.tinkerpop.blueprints.GraphFactory;
|
||||
import com.tinkerpop.blueprints.Vertex;
|
||||
|
||||
import edu.jhuapl.tinkerpop.cache.ElementCache;
|
||||
|
||||
public class ElementCacheTest {
|
||||
|
||||
@Test
|
||||
public void testElementCacheSize() throws Exception {
|
||||
AccumuloGraphConfiguration cfg = AccumuloGraphTestUtils
|
||||
.generateGraphConfig("elementCacheSize");
|
||||
Graph graph = GraphFactory.open(cfg.getConfiguration());
|
||||
|
||||
Vertex[] verts = new Vertex[10];
|
||||
for (int i = 0; i < verts.length; i++) {
|
||||
verts[i] = graph.addVertex(i);
|
||||
}
|
||||
|
||||
Edge[] edges = new Edge[9];
|
||||
for (int i = 0; i < edges.length; i++) {
|
||||
edges[i] = graph.addEdge(null,
|
||||
verts[0], verts[i+1], "edge");
|
||||
}
|
||||
|
||||
sizeTests(verts);
|
||||
sizeTests(edges);
|
||||
|
||||
graph.shutdown();
|
||||
}
|
||||
|
||||
private void sizeTests(Element[] elts) {
|
||||
ElementCache<Element> cache =
|
||||
new ElementCache<Element>(3, 120000);
|
||||
for (Element e : elts) {
|
||||
cache.cache(e);
|
||||
}
|
||||
|
||||
for (Element e : elts) {
|
||||
cache.cache(e);
|
||||
}
|
||||
|
||||
int total = 0;
|
||||
for (Element e : elts) {
|
||||
if (cache.retrieve(e.getId()) != null) {
|
||||
total++;
|
||||
}
|
||||
}
|
||||
assertTrue(total < elts.length);
|
||||
|
||||
cache.clear();
|
||||
for (Element e : elts) {
|
||||
assertNull(cache.retrieve(e.getId()));
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testElementCacheTimeout() throws Exception {
|
||||
AccumuloGraphConfiguration cfg = AccumuloGraphTestUtils
|
||||
.generateGraphConfig("elementCacheTimeout");
|
||||
Graph graph = GraphFactory.open(cfg.getConfiguration());
|
||||
|
||||
ElementCache<Element> cache =
|
||||
new ElementCache<Element>(10, 1000);
|
||||
|
||||
Vertex v1 = graph.addVertex(1);
|
||||
Vertex v2 = graph.addVertex(2);
|
||||
assertNull(cache.retrieve(1));
|
||||
assertNull(cache.retrieve(2));
|
||||
|
||||
cache.cache(v1);
|
||||
assertNotNull(cache.retrieve(v1.getId()));
|
||||
Thread.sleep(1500);
|
||||
assertNull(cache.retrieve(v1.getId()));
|
||||
|
||||
Edge e = graph.addEdge(null, v1, v2, "label");
|
||||
assertNull(cache.retrieve(e.getId()));
|
||||
|
||||
cache.cache(e);
|
||||
assertNotNull(cache.retrieve(e.getId()));
|
||||
Thread.sleep(1500);
|
||||
assertNull(cache.retrieve(e.getId()));
|
||||
|
||||
graph.shutdown();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,273 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop;
|
||||
|
||||
import static org.junit.Assert.*;
|
||||
|
||||
import org.junit.Test;
|
||||
|
||||
import com.google.common.collect.Sets;
|
||||
import com.tinkerpop.blueprints.Element;
|
||||
import com.tinkerpop.blueprints.Graph;
|
||||
import com.tinkerpop.blueprints.GraphFactory;
|
||||
|
||||
/**
|
||||
* Tests related to {@link Element}-based property
|
||||
* loading and caching.
|
||||
*/
|
||||
public class ElementPropertyCachingTest {
|
||||
|
||||
private static final int TIMEOUT = 300000;
|
||||
private static final String NON_CACHED = "noncached";
|
||||
private static final String CACHED = "cached";
|
||||
|
||||
@Test
|
||||
public void testCachingDisabled() {
|
||||
AccumuloGraphConfiguration cfg =
|
||||
AccumuloGraphTestUtils.generateGraphConfig("cachingDisabled");
|
||||
assertTrue(cfg.getPropertyCacheTimeout(null) <= 0);
|
||||
assertTrue(cfg.getPropertyCacheTimeout(NON_CACHED) <= 0);
|
||||
assertTrue(cfg.getPropertyCacheTimeout(CACHED) <= 0);
|
||||
|
||||
Graph graph = open(cfg);
|
||||
load(graph);
|
||||
|
||||
AccumuloVertex a = (AccumuloVertex) graph.getVertex("A");
|
||||
AccumuloVertex b = (AccumuloVertex) graph.getVertex("B");
|
||||
AccumuloVertex c = (AccumuloVertex) graph.getVertex("C");
|
||||
|
||||
assertEquals(null, a.getProperty(NON_CACHED));
|
||||
assertEquals(true, b.getProperty(NON_CACHED));
|
||||
assertEquals(null, c.getProperty(NON_CACHED));
|
||||
assertEquals(null, a.getProperty(CACHED));
|
||||
assertEquals(null, b.getProperty(CACHED));
|
||||
assertEquals(true, c.getProperty(CACHED));
|
||||
|
||||
assertEquals(null, a.getPropertyCache().get(NON_CACHED));
|
||||
assertEquals(null, b.getPropertyCache().get(NON_CACHED));
|
||||
assertEquals(null, c.getPropertyCache().get(NON_CACHED));
|
||||
assertEquals(null, a.getPropertyCache().get(CACHED));
|
||||
assertEquals(null, b.getPropertyCache().get(CACHED));
|
||||
assertEquals(null, c.getPropertyCache().get(CACHED));
|
||||
|
||||
assertEquals(Sets.newHashSet(), a.getPropertyCache().keySet());
|
||||
assertEquals(Sets.newHashSet(), b.getPropertyCache().keySet());
|
||||
assertEquals(Sets.newHashSet(), c.getPropertyCache().keySet());
|
||||
|
||||
a.removeProperty(NON_CACHED);
|
||||
b.removeProperty(NON_CACHED);
|
||||
c.removeProperty(NON_CACHED);
|
||||
a.removeProperty(CACHED);
|
||||
b.removeProperty(CACHED);
|
||||
c.removeProperty(CACHED);
|
||||
|
||||
assertEquals(null, a.getProperty(NON_CACHED));
|
||||
assertEquals(null, b.getProperty(NON_CACHED));
|
||||
assertEquals(null, c.getProperty(NON_CACHED));
|
||||
assertEquals(null, a.getProperty(CACHED));
|
||||
assertEquals(null, b.getProperty(CACHED));
|
||||
assertEquals(null, c.getProperty(CACHED));
|
||||
|
||||
assertEquals(null, a.getPropertyCache().get(NON_CACHED));
|
||||
assertEquals(null, b.getPropertyCache().get(NON_CACHED));
|
||||
assertEquals(null, c.getPropertyCache().get(NON_CACHED));
|
||||
assertEquals(null, a.getPropertyCache().get(CACHED));
|
||||
assertEquals(null, b.getPropertyCache().get(CACHED));
|
||||
assertEquals(null, c.getPropertyCache().get(CACHED));
|
||||
|
||||
assertEquals(Sets.newHashSet(), a.getPropertyCache().keySet());
|
||||
assertEquals(Sets.newHashSet(), b.getPropertyCache().keySet());
|
||||
assertEquals(Sets.newHashSet(), c.getPropertyCache().keySet());
|
||||
|
||||
graph.shutdown();
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testSpecificCaching() {
|
||||
AccumuloGraphConfiguration cfg =
|
||||
AccumuloGraphTestUtils.generateGraphConfig("getProperty");
|
||||
cfg.setPropertyCacheTimeout(CACHED, TIMEOUT);
|
||||
|
||||
assertTrue(cfg.getPropertyCacheTimeout(null) <= 0);
|
||||
assertTrue(cfg.getPropertyCacheTimeout(NON_CACHED) <= 0);
|
||||
assertEquals(TIMEOUT, cfg.getPropertyCacheTimeout(CACHED));
|
||||
|
||||
Graph graph = open(cfg);
|
||||
load(graph);
|
||||
|
||||
AccumuloVertex a = (AccumuloVertex) graph.getVertex("A");
|
||||
AccumuloVertex b = (AccumuloVertex) graph.getVertex("B");
|
||||
AccumuloVertex c = (AccumuloVertex) graph.getVertex("C");
|
||||
|
||||
assertEquals(null, a.getProperty(NON_CACHED));
|
||||
assertEquals(true, b.getProperty(NON_CACHED));
|
||||
assertEquals(null, c.getProperty(NON_CACHED));
|
||||
assertEquals(null, a.getProperty(CACHED));
|
||||
assertEquals(null, b.getProperty(CACHED));
|
||||
assertEquals(true, c.getProperty(CACHED));
|
||||
|
||||
assertEquals(null, a.getPropertyCache().get(NON_CACHED));
|
||||
assertEquals(null, b.getPropertyCache().get(NON_CACHED));
|
||||
assertEquals(null, c.getPropertyCache().get(NON_CACHED));
|
||||
assertEquals(null, a.getPropertyCache().get(CACHED));
|
||||
assertEquals(null, b.getPropertyCache().get(CACHED));
|
||||
assertEquals(true, c.getPropertyCache().get(CACHED));
|
||||
|
||||
assertEquals(Sets.newHashSet(), a.getPropertyCache().keySet());
|
||||
assertEquals(Sets.newHashSet(), b.getPropertyCache().keySet());
|
||||
assertEquals(Sets.newHashSet(CACHED), c.getPropertyCache().keySet());
|
||||
|
||||
a.removeProperty(NON_CACHED);
|
||||
b.removeProperty(NON_CACHED);
|
||||
c.removeProperty(NON_CACHED);
|
||||
a.removeProperty(CACHED);
|
||||
b.removeProperty(CACHED);
|
||||
c.removeProperty(CACHED);
|
||||
|
||||
assertEquals(null, a.getProperty(NON_CACHED));
|
||||
assertEquals(null, b.getProperty(NON_CACHED));
|
||||
assertEquals(null, c.getProperty(NON_CACHED));
|
||||
assertEquals(null, a.getProperty(CACHED));
|
||||
assertEquals(null, b.getProperty(CACHED));
|
||||
assertEquals(null, c.getProperty(CACHED));
|
||||
|
||||
assertEquals(null, a.getPropertyCache().get(NON_CACHED));
|
||||
assertEquals(null, b.getPropertyCache().get(NON_CACHED));
|
||||
assertEquals(null, c.getPropertyCache().get(NON_CACHED));
|
||||
assertEquals(null, a.getPropertyCache().get(CACHED));
|
||||
assertEquals(null, b.getPropertyCache().get(CACHED));
|
||||
assertEquals(null, c.getPropertyCache().get(CACHED));
|
||||
|
||||
assertEquals(Sets.newHashSet(), a.getPropertyCache().keySet());
|
||||
assertEquals(Sets.newHashSet(), b.getPropertyCache().keySet());
|
||||
assertEquals(Sets.newHashSet(), c.getPropertyCache().keySet());
|
||||
|
||||
graph.shutdown();
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testAllCaching() {
|
||||
AccumuloGraphConfiguration cfg =
|
||||
AccumuloGraphTestUtils.generateGraphConfig("setProperty");
|
||||
cfg.setPropertyCacheTimeout(null, TIMEOUT);
|
||||
cfg.setPropertyCacheTimeout(CACHED, TIMEOUT);
|
||||
|
||||
assertEquals(TIMEOUT, cfg.getPropertyCacheTimeout(null));
|
||||
assertEquals(TIMEOUT, cfg.getPropertyCacheTimeout(NON_CACHED));
|
||||
assertEquals(TIMEOUT, cfg.getPropertyCacheTimeout(CACHED));
|
||||
|
||||
Graph graph = open(cfg);
|
||||
load(graph);
|
||||
|
||||
AccumuloVertex a = (AccumuloVertex) graph.getVertex("A");
|
||||
AccumuloVertex b = (AccumuloVertex) graph.getVertex("B");
|
||||
AccumuloVertex c = (AccumuloVertex) graph.getVertex("C");
|
||||
|
||||
assertEquals(null, a.getProperty(NON_CACHED));
|
||||
assertEquals(true, b.getProperty(NON_CACHED));
|
||||
assertEquals(null, c.getProperty(NON_CACHED));
|
||||
assertEquals(null, a.getProperty(CACHED));
|
||||
assertEquals(null, b.getProperty(CACHED));
|
||||
assertEquals(true, c.getProperty(CACHED));
|
||||
|
||||
assertEquals(null, a.getPropertyCache().get(NON_CACHED));
|
||||
assertEquals(true, b.getPropertyCache().get(NON_CACHED));
|
||||
assertEquals(null, c.getPropertyCache().get(NON_CACHED));
|
||||
assertEquals(null, a.getPropertyCache().get(CACHED));
|
||||
assertEquals(null, b.getPropertyCache().get(CACHED));
|
||||
assertEquals(true, c.getPropertyCache().get(CACHED));
|
||||
|
||||
assertEquals(Sets.newHashSet(), a.getPropertyCache().keySet());
|
||||
assertEquals(Sets.newHashSet(NON_CACHED), b.getPropertyCache().keySet());
|
||||
assertEquals(Sets.newHashSet(CACHED), c.getPropertyCache().keySet());
|
||||
|
||||
a.removeProperty(NON_CACHED);
|
||||
b.removeProperty(NON_CACHED);
|
||||
c.removeProperty(NON_CACHED);
|
||||
a.removeProperty(CACHED);
|
||||
b.removeProperty(CACHED);
|
||||
c.removeProperty(CACHED);
|
||||
|
||||
assertEquals(null, a.getProperty(NON_CACHED));
|
||||
assertEquals(null, b.getProperty(NON_CACHED));
|
||||
assertEquals(null, c.getProperty(NON_CACHED));
|
||||
assertEquals(null, a.getProperty(CACHED));
|
||||
assertEquals(null, b.getProperty(CACHED));
|
||||
assertEquals(null, c.getProperty(CACHED));
|
||||
|
||||
assertEquals(null, a.getPropertyCache().get(NON_CACHED));
|
||||
assertEquals(null, b.getPropertyCache().get(NON_CACHED));
|
||||
assertEquals(null, c.getPropertyCache().get(NON_CACHED));
|
||||
assertEquals(null, a.getPropertyCache().get(CACHED));
|
||||
assertEquals(null, b.getPropertyCache().get(CACHED));
|
||||
assertEquals(null, c.getPropertyCache().get(CACHED));
|
||||
|
||||
assertEquals(Sets.newHashSet(), a.getPropertyCache().keySet());
|
||||
assertEquals(Sets.newHashSet(), b.getPropertyCache().keySet());
|
||||
assertEquals(Sets.newHashSet(), c.getPropertyCache().keySet());
|
||||
|
||||
graph.shutdown();
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testPreloadAllProperties() {
|
||||
AccumuloGraphConfiguration cfg =
|
||||
AccumuloGraphTestUtils.generateGraphConfig("preloadAllProperties");
|
||||
cfg.setPropertyCacheTimeout(null, TIMEOUT);
|
||||
cfg.setPreloadAllProperties(true);
|
||||
|
||||
Graph graph = open(cfg);
|
||||
|
||||
AccumuloVertex v = (AccumuloVertex) graph.addVertex("V");
|
||||
v.setProperty(NON_CACHED, true);
|
||||
v.setProperty(CACHED, true);
|
||||
|
||||
v = (AccumuloVertex) graph.getVertex("V");
|
||||
assertEquals(true, v.getPropertyInMemory(NON_CACHED));
|
||||
assertEquals(true, v.getPropertyInMemory(CACHED));
|
||||
|
||||
graph.shutdown();
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testPreloadSomeProperties() {
|
||||
AccumuloGraphConfiguration cfg =
|
||||
AccumuloGraphTestUtils.generateGraphConfig("preloadSomeProperties");
|
||||
cfg.setPropertyCacheTimeout(null, TIMEOUT);
|
||||
cfg.setPreloadedProperties(new String[]{CACHED});
|
||||
|
||||
Graph graph = open(cfg);
|
||||
|
||||
AccumuloVertex v = (AccumuloVertex) graph.addVertex("V");
|
||||
v.setProperty(NON_CACHED, true);
|
||||
v.setProperty(CACHED, true);
|
||||
|
||||
v = (AccumuloVertex) graph.getVertex("V");
|
||||
assertEquals(null, v.getPropertyInMemory(NON_CACHED));
|
||||
assertEquals(true, v.getPropertyInMemory(CACHED));
|
||||
|
||||
graph.shutdown();
|
||||
}
|
||||
|
||||
private static Graph open(AccumuloGraphConfiguration cfg) {
|
||||
return GraphFactory.open(cfg);
|
||||
}
|
||||
|
||||
private static void load(Graph graph) {
|
||||
graph.addVertex("A");
|
||||
graph.addVertex("B").setProperty(NON_CACHED, true);
|
||||
graph.addVertex("C").setProperty(CACHED, true);
|
||||
}
|
||||
}
|
||||
@@ -4,11 +4,16 @@ import com.tinkerpop.blueprints.Graph;
|
||||
import com.tinkerpop.blueprints.GraphFactory;
|
||||
|
||||
public class ExtendedAccumuloGraphTest extends AccumuloGraphTest {
|
||||
|
||||
@Override
|
||||
public Graph generateGraph(String graphDirectoryName) {
|
||||
AccumuloGraphConfiguration cfg = AccumuloGraphTestUtils.generateGraphConfig(graphDirectoryName);
|
||||
cfg.setLruMaxCapacity(20).setPreloadedProperties(new String[] {"name"}).setPreloadedEdgeLabels(new String[] {"knows"}).setPropertyCacheTimeout("name",100000);
|
||||
cfg.setEdgeCacheParams(20, 30000)
|
||||
.setPreloadedProperties(new String[] {"name"})
|
||||
.setPreloadedEdgeLabels(new String[] {"knows"})
|
||||
.setPropertyCacheTimeout("name", 100000);
|
||||
testGraphName.set(graphDirectoryName);
|
||||
return GraphFactory.open(cfg.getConfiguration());
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
107
src/test/java/edu/jhuapl/tinkerpop/ExtendedElementTest.java
Normal file
107
src/test/java/edu/jhuapl/tinkerpop/ExtendedElementTest.java
Normal file
@@ -0,0 +1,107 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop;
|
||||
|
||||
import static org.junit.Assert.*;
|
||||
|
||||
import org.junit.Test;
|
||||
|
||||
import com.tinkerpop.blueprints.Graph;
|
||||
import com.tinkerpop.blueprints.GraphFactory;
|
||||
import com.tinkerpop.blueprints.Vertex;
|
||||
|
||||
/**
|
||||
* Tests related to implementation-specific elements.
|
||||
*/
|
||||
public class ExtendedElementTest {
|
||||
|
||||
private Graph makeGraph(AccumuloGraphConfiguration cfg) {
|
||||
return GraphFactory.open(cfg.getConfiguration());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testExistenceChecks() throws Exception {
|
||||
AccumuloGraphConfiguration cfg =
|
||||
AccumuloGraphTestUtils.generateGraphConfig("yesExistenceChecks");
|
||||
Graph graph = makeGraph(cfg);
|
||||
|
||||
String id;
|
||||
|
||||
|
||||
id = "doubleAdd";
|
||||
assertNotNull(graph.addVertex(id));
|
||||
try {
|
||||
graph.addVertex(id);
|
||||
fail();
|
||||
} catch (Exception e) { }
|
||||
|
||||
Vertex vAdd = graph.getVertex(id);
|
||||
assertNotNull(vAdd);
|
||||
graph.removeVertex(vAdd);
|
||||
assertNull(graph.getVertex(id));
|
||||
|
||||
|
||||
id = "doubleRemove";
|
||||
Vertex vRemove = graph.addVertex(id);
|
||||
assertNotNull(vRemove);
|
||||
graph.removeVertex(vRemove);
|
||||
try {
|
||||
graph.removeVertex(vRemove);
|
||||
fail();
|
||||
} catch (Exception e) { }
|
||||
assertNull(graph.getVertex(id));
|
||||
|
||||
|
||||
id = "notExist";
|
||||
assertNull(graph.getVertex(id));
|
||||
|
||||
|
||||
graph.shutdown();
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testSkipExistenceChecks() throws Exception {
|
||||
AccumuloGraphConfiguration cfg =
|
||||
AccumuloGraphTestUtils.generateGraphConfig("skipExistenceChecks");
|
||||
cfg.setSkipExistenceChecks(true);
|
||||
Graph graph = makeGraph(cfg);
|
||||
|
||||
String id;
|
||||
|
||||
id = "doubleAdd";
|
||||
assertNotNull(graph.addVertex(id));
|
||||
assertNotNull(graph.addVertex(id));
|
||||
Vertex vAdd = graph.getVertex(id);
|
||||
assertNotNull(vAdd);
|
||||
graph.removeVertex(vAdd);
|
||||
assertNotNull(graph.getVertex(id));
|
||||
|
||||
|
||||
id = "doubleRemove";
|
||||
Vertex vRemove = graph.addVertex(id);
|
||||
assertNotNull(vRemove);
|
||||
graph.removeVertex(vRemove);
|
||||
assertNotNull(graph.getVertex(id));
|
||||
// MDL 24 Dec 2014: removeVertex still does checks.
|
||||
//graph.removeVertex(vRemove);
|
||||
//assertNotNull(graph.getVertex(id));
|
||||
|
||||
|
||||
id = "notExist";
|
||||
assertNotNull(graph.getVertex(id));
|
||||
|
||||
graph.shutdown();
|
||||
}
|
||||
}
|
||||
23
src/test/java/edu/jhuapl/tinkerpop/MockInstanceTest.java
Normal file
23
src/test/java/edu/jhuapl/tinkerpop/MockInstanceTest.java
Normal file
@@ -0,0 +1,23 @@
|
||||
package edu.jhuapl.tinkerpop;
|
||||
|
||||
import org.apache.commons.configuration.Configuration;
|
||||
|
||||
import com.tinkerpop.blueprints.Graph;
|
||||
import com.tinkerpop.blueprints.GraphFactory;
|
||||
|
||||
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration.InstanceType;
|
||||
|
||||
/**
|
||||
* Run the test suite for a Mock instance type.
|
||||
*/
|
||||
public class MockInstanceTest extends AccumuloGraphTest {
|
||||
|
||||
@Override
|
||||
public Graph generateGraph(String graphDirectoryName) {
|
||||
Configuration cfg = new AccumuloGraphConfiguration()
|
||||
.setInstanceType(InstanceType.Mock)
|
||||
.setGraphName(graphDirectoryName);
|
||||
testGraphName.set(graphDirectoryName);
|
||||
return GraphFactory.open(cfg);
|
||||
}
|
||||
}
|
||||
87
src/test/java/edu/jhuapl/tinkerpop/PropertyCacheTest.java
Normal file
87
src/test/java/edu/jhuapl/tinkerpop/PropertyCacheTest.java
Normal file
@@ -0,0 +1,87 @@
|
||||
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package edu.jhuapl.tinkerpop;
|
||||
|
||||
import static org.junit.Assert.*;
|
||||
|
||||
import org.junit.Test;
|
||||
|
||||
import edu.jhuapl.tinkerpop.cache.PropertyCache;
|
||||
|
||||
/**
|
||||
* Test the {@link PropertyCache} object.
|
||||
*/
|
||||
public class PropertyCacheTest {
|
||||
|
||||
@Test
|
||||
public void testUncachedProperty() throws Exception {
|
||||
AccumuloGraphConfiguration cfg =
|
||||
AccumuloGraphTestUtils.generateGraphConfig("uncached");
|
||||
PropertyCache cache = new PropertyCache(cfg);
|
||||
cache.put("K1", "V1");
|
||||
cache.put("K2", "V2");
|
||||
cache.put("K3", "V3");
|
||||
assertNull(cache.get("K1"));
|
||||
assertNull(cache.get("K2"));
|
||||
assertNull(cache.get("K3"));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testDefaultCachedProperty() throws Exception {
|
||||
AccumuloGraphConfiguration cfg =
|
||||
AccumuloGraphTestUtils.generateGraphConfig("defaultCached");
|
||||
cfg.setPropertyCacheTimeout(null, 1000);
|
||||
|
||||
PropertyCache cache = new PropertyCache(cfg);
|
||||
cache.put("K1", "V1");
|
||||
assertEquals("V1", cache.get("K1"));
|
||||
Thread.sleep(1500);
|
||||
assertNull(cache.get("K1"));
|
||||
|
||||
cache.put("K2", "V2");
|
||||
assertEquals("V2", cache.get("K2"));
|
||||
cache.remove("K2");
|
||||
assertNull(cache.get("K2"));
|
||||
|
||||
cache.put("K3", "V3");
|
||||
cache.clear();
|
||||
assertNull(cache.get("K3"));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testSpecificCachedProperty() throws Exception {
|
||||
AccumuloGraphConfiguration cfg =
|
||||
AccumuloGraphTestUtils.generateGraphConfig("specificCached");
|
||||
cfg.setPropertyCacheTimeout(null, 1000);
|
||||
cfg.setPropertyCacheTimeout("long", 2000);
|
||||
cfg.setPropertyCacheTimeout("longer", 3000);
|
||||
|
||||
PropertyCache cache = new PropertyCache(cfg);
|
||||
cache.put("default", "V1");
|
||||
cache.put("long", "V2");
|
||||
cache.put("longer", "V3");
|
||||
|
||||
assertEquals("V1", cache.get("default"));
|
||||
assertEquals("V2", cache.get("long"));
|
||||
assertEquals("V3", cache.get("longer"));
|
||||
|
||||
Thread.sleep(1500);
|
||||
assertNull(cache.get("default"));
|
||||
Thread.sleep(1000);
|
||||
assertNull(cache.get("long"));
|
||||
Thread.sleep(1000);
|
||||
assertNull(cache.get("longer"));
|
||||
}
|
||||
}
|
||||
@@ -136,35 +136,50 @@ public class InputFormatsTest {
|
||||
@Test
|
||||
public void testVertexInputMap() throws Exception {
|
||||
final String INSTANCE_NAME = "_mapreduce_instance";
|
||||
final String TEST_TABLE_1 = "_mapreduce_table_1";
|
||||
final String TEST_TABLE_NAME = "_mapreduce_table_vertexInputMap";
|
||||
|
||||
if (!System.getProperty("os.name").startsWith("Windows")) {
|
||||
Graph g = GraphFactory.open(new AccumuloGraphConfiguration().setInstanceName(INSTANCE_NAME).setUser("root").setPassword("".getBytes())
|
||||
.setGraphName(TEST_TABLE_1).setInstanceType(InstanceType.Mock).setCreate(true).getConfiguration());
|
||||
Graph g = GraphFactory.open(new AccumuloGraphConfiguration().setInstanceName(INSTANCE_NAME)
|
||||
.setUser("root").setPassword("".getBytes())
|
||||
.setGraphName(TEST_TABLE_NAME).setInstanceType(InstanceType.Mock)
|
||||
.setCreate(true).getConfiguration());
|
||||
|
||||
for (int i = 0; i < 100; i++) {
|
||||
g.addVertex(i + "");
|
||||
}
|
||||
assertEquals(0, MRTester.main(new String[] {"root", "", TEST_TABLE_1, INSTANCE_NAME, "false"}));
|
||||
|
||||
assertEquals(0, MRTester.main(new String[]{"root", "",
|
||||
TEST_TABLE_NAME, INSTANCE_NAME, "false"}));
|
||||
assertNull(e1);
|
||||
assertNull(e2);
|
||||
|
||||
g.shutdown();
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testEdgeInputMap() throws Exception {
|
||||
final String INSTANCE_NAME = "_mapreduce_instance";
|
||||
final String TEST_TABLE_1 = "_mapreduce_table_1";
|
||||
final String TEST_TABLE_NAME = "_mapreduce_table_edgeInputMap";
|
||||
|
||||
if (!System.getProperty("os.name").startsWith("Windows")) {
|
||||
Graph g = GraphFactory.open(new AccumuloGraphConfiguration().setInstanceName(INSTANCE_NAME).setUser("root").setPassword("".getBytes())
|
||||
.setGraphName(TEST_TABLE_1).setInstanceType(InstanceType.Mock).autoFlush(true).setCreate(true).getConfiguration());
|
||||
for (int i = 0; i < 100; i++) {
|
||||
g.addEdge(null, g.addVertex(i + ""), g.addVertex(i + "a"), "knows");
|
||||
Graph g = GraphFactory.open(new AccumuloGraphConfiguration().setInstanceName(INSTANCE_NAME)
|
||||
.setUser("root").setPassword("".getBytes())
|
||||
.setGraphName(TEST_TABLE_NAME).setInstanceType(InstanceType.Mock)
|
||||
.setAutoFlush(true).setCreate(true).getConfiguration());
|
||||
|
||||
for (int i = 0; i < 100; i++) {
|
||||
Vertex v1 = g.addVertex(i+"");
|
||||
Vertex v2 = g.addVertex(i+"a");
|
||||
g.addEdge(null, v1, v2, "knows");
|
||||
}
|
||||
assertEquals(0, MRTester.main(new String[] {"root", "", TEST_TABLE_1, INSTANCE_NAME, "true"}));
|
||||
|
||||
assertEquals(0, MRTester.main(new String[]{"root", "",
|
||||
TEST_TABLE_NAME, INSTANCE_NAME, "true"}));
|
||||
assertNull(e1);
|
||||
assertNull(e2);
|
||||
|
||||
g.shutdown();
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
93
table-structure.md
Normal file
93
table-structure.md
Normal file
@@ -0,0 +1,93 @@
|
||||
AccumuloGraph Table Schema
|
||||
==========================
|
||||
|
||||
AccumuloGraph uses a number of backing tables for data storage.
|
||||
This file documents the structure and organization of these
|
||||
tables. The tables and records are structured in such a way as
|
||||
to implement the Blueprints operations in terms of efficient Accumulo
|
||||
operations, i.e., prefix searches, contiguous scans, and
|
||||
batch operations.
|
||||
For our purposes, Accumulo entries consist of four fields:
|
||||
row (R), column family (CF), column qualifier (CQ), and value (V).
|
||||
For more information on Accumulo's schema and how to
|
||||
access records efficiently, see the
|
||||
[Accumulo documentation](https://accumulo.apache.org/1.5/accumulo_user_manual.html).
|
||||
|
||||
The tables used by AccumuloGraph are prefixed by the configured graph
|
||||
name and can be classed as element/property tables, and
|
||||
index tables. Their structure is discussed below.
|
||||
|
||||
Elements and properties
|
||||
-----------------------
|
||||
|
||||
Vertex and edge information, along with their properties, are stored
|
||||
in the *graphname*\_vertex and *graphname*\_edge tables respectively.
|
||||
|
||||
First, an entry declares the existence of a vertex in the vertex table.
|
||||
|
||||
| R | CF | CQ | V |
|
||||
|---|----|----|---|
|
||||
| *vertex_id* | `_LABEL_` | `_EXISTS_`| *[empty]* |
|
||||
|
||||
A similar entry declares the existence of an edge in the edge table.
|
||||
|
||||
| R | CF | CQ | V |
|
||||
|---|----|----|---|
|
||||
| *edge_id* | `_LABEL_` | *in_vertex_id*`_DELIM_`*out_vertex_id* | *edge_label* |
|
||||
|
||||
When adding an edge, additional entries are stored in the
|
||||
vertex table for each endpoint of the edge. These facilitate the
|
||||
`Vertex.getEdge` and `Vertex.getVertex` operations.
|
||||
|
||||
| R | CF | CQ | V |
|
||||
|---|----|----|---|
|
||||
| *in_vertex_id* | `_IN_EDGE_` | *out_vertex_id*`_DELIM_`*edge_id* | *edge_label* |
|
||||
| *out_vertex_id* | `_OUT_EDGE_` | *in_vertex_id*`_DELIM_`*edge_id* | *edge_label* |
|
||||
|
||||
Finally, vertex and edge properties are stored in their respective
|
||||
tables. Entry formats are the same for both vertices and edges.
|
||||
Note that property values are serialized such that their type
|
||||
can be deduced when deserializing.
|
||||
|
||||
| R | CF | CQ | V |
|
||||
|---|----|----|---|
|
||||
| *element_id* | *property_key* | *[empty]* | *property_value* |
|
||||
|
||||
Indexes
|
||||
-------
|
||||
|
||||
Several tables store index-related information,
|
||||
including index value tables that store index
|
||||
property keys and values, and index metadata tables
|
||||
that store information about what indexes exist
|
||||
and what properties are indexed.
|
||||
|
||||
For `KeyIndexableGraph`, index value tables
|
||||
include *graphname*\_vertex\_key\_index
|
||||
and *graphname*\_edge\_key\_index for vertex
|
||||
and edge properties, respectively.
|
||||
For `IndexableGraph`, index value tables are
|
||||
named *graphname*\_index\_*indexname*,
|
||||
where *indexname* is the index name.
|
||||
The entry formats in all these tables are the same:
|
||||
|
||||
| R | CF | CQ | V |
|
||||
|---|----|----|---|
|
||||
| *property_value* | *property_key* | *element_id* | *[empty]* |
|
||||
|
||||
Property values are serialized in the same way as above.
|
||||
|
||||
In addition to the index value tables, an index metadata table,
|
||||
*graphname*\_index\_metadata, stores index information. For
|
||||
`KeyIndexableGraph`, records in this table enumerate the property keys
|
||||
that are indexed.
|
||||
|
||||
| R | CF | CQ | V |
|
||||
|---|----|----|---|
|
||||
| *property_key* | `_INDEX_KEY_` | *element_class* | *[empty]* |
|
||||
|
||||
For `IndexableGraph`, records enumerate the existing indexes.
|
||||
|
||||
| R | CF | CQ | V |
|
||||
|---|----|----|---|
|
||||
| *index_name* | `_INDEX_NAME_` | *element_class* | *[empty]* |
|
||||
Reference in New Issue
Block a user