135 Commits

Author SHA1 Message Date
Ryan Webb
9abbc25f4e forgot to do zookeeper 2015-02-20 15:00:10 -05:00
Ryan Webb
d717030169 Workaround just to see if tests run on mini 2015-02-20 14:32:37 -05:00
Ryan Webb
a2e0a9b6f9 Fix for missing variables
go travisci go
2015-02-20 14:27:33 -05:00
Ryan Webb
fffc731507 Updated tests to use mini.
Does not build on windows -- will fix after fixing on CI build
2015-02-20 14:19:06 -05:00
Ryan Webb
1f630eb3ef Next Snapshot 2015-02-20 12:32:25 -05:00
Ryan Webb
718258e828 2.1 Release 2015-02-20 12:32:04 -05:00
Ryan Webb
99e01d8608 Added changelog 2015-02-20 12:28:35 -05:00
mikelieberman
a42ae0a521 Merge pull request #108 from JHUAPL/connector-reuse
Added functionality to reuse Connector, and prohibit configuration changes. Addresses #85 - a connector pool is not used, but a single instance is instead.
2015-02-09 17:29:37 -05:00
Michael Lieberman
00056f6513 Added functionality to reuse Connector, and prohibit configuration changes after use 2015-02-09 17:20:48 -05:00
mikelieberman
8df511c2d1 Merge pull request #107 from JHUAPL/preload-all-properties
Preload all properties
2015-02-09 16:17:45 -05:00
Michael Lieberman
ec13b05459 Added "preload all" functionality and unit tests for same 2015-02-09 16:08:10 -05:00
Michael Lieberman
37157a0093 Added unit test for property preloading 2015-02-09 15:48:45 -05:00
Michael Lieberman
438830f5f4 Added setPreloadAllProperties to AccumuloGraphConfiguration 2015-02-09 15:06:36 -05:00
mikelieberman
1018d48382 Merge pull request #106 from JHUAPL/merged-metadata-tables
Merged metadata tables
2015-02-09 13:52:30 -05:00
mikelieberman
30468ed2f3 Updated table structure documentation to match metadata table changes 2015-02-09 13:43:27 -05:00
Michael Lieberman
6faac76ba9 Merge metadata tables into one table 2015-02-09 13:35:06 -05:00
mikelieberman
bb6f9e80a7 Merge pull request #105 from JHUAPL/get-vertices-id-range
Get vertices id range
2015-02-06 18:47:53 -05:00
Michael Lieberman
56929af885 Added documentation and unit test for getVerticesInRange 2015-02-06 18:42:13 -05:00
Michael Lieberman
89976c2c86 Add getVerticesInRange functionality 2015-02-06 18:16:40 -05:00
mikelieberman
816dffe141 Merge pull request #104 from JHUAPL/table-structure-documentation
Table structure documentation
2015-02-06 17:58:13 -05:00
mikelieberman
a7035e259a Updated table structure documentation 2015-02-06 17:55:34 -05:00
Michael Lieberman
f745ff23fb Initial commit of table structure documentation; still needs work 2015-02-06 15:24:07 -05:00
Michael Lieberman
93c9c37936 Replace printStackTrace with exceptions 2015-02-05 17:27:34 -05:00
Michael Lieberman
c08bb5058f FindBugs tweaks 2015-01-28 15:52:58 -05:00
webbrl1
dcea092c40 Fixed High and Medium Coverity bugs. Fixed a few Low bugs 2015-01-28 14:25:11 -05:00
webbrl1
f9bb666e4b Release and updated version 2015-01-28 13:26:09 -05:00
Ryan Webb
834e2cb2aa Merge pull request #99 from JHUAPL/table-wrappers
Table wrappers
2015-01-28 13:03:55 -05:00
Michael Lieberman
f72b1d2f2f Use mutator for adding edge in bulk ingester 2015-01-27 16:38:21 -05:00
Michael Lieberman
5dd5c978a7 Convert PropertyBuilder to use Mutators 2015-01-27 16:26:55 -05:00
Michael Lieberman
bca5bf4474 Use mutator for bulk ingester addProperty 2015-01-27 16:14:13 -05:00
Michael Lieberman
365dc740d7 Bulk ingester add vertex using mutator 2015-01-27 16:08:52 -05:00
Michael Lieberman
d5de29c000 Move createIndex to NamedIndexListTableWrapper 2015-01-27 14:32:55 -05:00
Michael Lieberman
0b2ec50eeb Move createKeyIndex to appropriate wrappers
Some other code cleanup
2015-01-27 14:22:17 -05:00
Michael Lieberman
8a36bc7930 Get rid of graph member from GlobalInstances 2015-01-27 11:44:15 -05:00
Michael Lieberman
38b22df42f Fix AutoIndexTest unit tests and bug fixes 2015-01-27 11:18:12 -05:00
Michael Lieberman
1123268b37 Use TableOperations for isEmpty 2015-01-26 17:17:43 -05:00
Michael Lieberman
be81489672 Clean up unused functionality 2015-01-26 17:07:48 -05:00
Michael Lieberman
37774d60a1 Fix AccumuloVertex.remove, and probably another bug with AccumuloEdge.remove 2015-01-26 16:56:03 -05:00
Michael Lieberman
e2c5af3752 Added delete vertex functionality to VertexTableWrapper 2015-01-26 15:30:37 -05:00
Michael Lieberman
9afc558d95 Move getIndex functionality to NamedIndexListTableWrapper 2015-01-26 12:06:24 -05:00
Michael Lieberman
a2502f5bb1 Move getIndices functionality to NamedIndexListTableWrapper 2015-01-26 11:56:02 -05:00
Michael Lieberman
49a5a77d1d Refactor table wrapper names to avoid confusion 2015-01-26 10:31:00 -05:00
Michael Lieberman
53f7ed4025 Get rid of old non-useful documentation 2015-01-26 09:55:10 -05:00
Michael Lieberman
9edf2ff66e Refactoring 2015-01-23 16:38:29 -05:00
Michael Lieberman
49d4463cc5 Cleanup 2015-01-23 14:44:39 -05:00
Michael Lieberman
37593c13b4 Refactor edge.remove to use existing rather than new functionality 2015-01-23 14:38:23 -05:00
Michael Lieberman
7bf543e502 Move removeEdge functionality into AccumuloEdge
Other cleanup
2015-01-23 14:22:20 -05:00
Michael Lieberman
43a3293ef0 Clean up removeEdge 2015-01-23 14:11:54 -05:00
Michael Lieberman
6fffd094f3 Remove 'cacheProperty' methods from AccumuloElement 2015-01-23 13:28:11 -05:00
Michael Lieberman
b60f855a5e Connect index-clearing code to AccumuloIndex
Other cleanup
2015-01-21 17:51:33 -05:00
Michael Lieberman
832d597f0a Add removeElementFromIndex to IndexTableWrapper 2015-01-21 17:37:36 -05:00
Michael Lieberman
7fb9cce693 Cleanup 2015-01-21 13:47:44 -05:00
Michael Lieberman
128b40346a Cleanup 2015-01-21 13:42:37 -05:00
Michael Lieberman
eacf44c755 Refactor AccumuloIndex put into IndexTableWrapper 2015-01-21 13:37:17 -05:00
Michael Lieberman
e59c590eae Move get functionality out of AccumuloIndex to IndexTableWrapper 2015-01-21 13:24:00 -05:00
Michael Lieberman
2dd9e66cf2 Use indexWrapper for AccumuloIndex.remove 2015-01-21 12:03:17 -05:00
Michael Lieberman
3897998fca Use index parser in AccumuloIndex 2015-01-21 12:01:37 -05:00
Michael Lieberman
a9d8ff7570 Added NamedIndexTableWrapper
Cleanup
2015-01-21 11:54:03 -05:00
Michael Lieberman
7e79174b1d Cleanup 2015-01-21 11:17:46 -05:00
Michael Lieberman
df0824f7de Cleanup 2015-01-20 17:04:35 -05:00
Michael Lieberman
864cee9312 Cleanup and documentation 2015-01-20 16:42:43 -05:00
Michael Lieberman
2544ac0c30 Move index-based getEdges to EdgeIndexTableWrapper 2015-01-20 16:36:07 -05:00
Michael Lieberman
16b7284f08 Move index-based getVertices to VertexIndexTableWrapper 2015-01-20 16:26:33 -05:00
Michael Lieberman
f0bd6b8cd6 Add index entry parsers 2015-01-20 16:04:23 -05:00
Michael Lieberman
379099adc2 Cleanup 2015-01-20 15:32:11 -05:00
Michael Lieberman
f0a23e3848 Move remote property from index functionality to IndexTableWrapper 2015-01-20 15:31:42 -05:00
Michael Lieberman
ad6ef0c2f1 Warnings cleanup 2015-01-20 15:27:20 -05:00
Michael Lieberman
5632417f30 Add IndexValueMutator mutator 2015-01-20 15:23:57 -05:00
Michael Lieberman
c17693968c Cleanup 2015-01-20 14:43:09 -05:00
Michael Lieberman
c66d86d123 Move setPropertyForIndex to IndexTableWrapper 2015-01-20 14:41:15 -05:00
Michael Lieberman
36ca8641ef Cleanup 2015-01-20 14:21:29 -05:00
Michael Lieberman
5032b72803 Add property index tables to globals 2015-01-20 13:52:26 -05:00
Michael Lieberman
c90a379e2b Rename tables to make their purpose clearer and other cleanup 2015-01-20 13:46:16 -05:00
Michael Lieberman
9832b92d7b Move getIndexedKeys to KeyMetadataTableWrapper
Also some other cleanup
2015-01-19 19:04:54 -05:00
Michael Lieberman
712420c2c5 Use IndexMetadataTableWrapper for index metadata
Other cleanup
2015-01-19 18:25:28 -05:00
Michael Lieberman
2c08dc149e Use same table format for key metadata and index metadata 2015-01-19 18:09:36 -05:00
Michael Lieberman
484e78f080 Documentation and cleanup 2015-01-19 17:57:13 -05:00
Michael Lieberman
2b3f07688d Several updates
Created key index metadata wrapper and moved some functionality there
Changed Mutator to an interface rather than abstract class
2015-01-19 17:42:37 -05:00
Michael Lieberman
ff17e6a5af Tweaks and cleanup 2015-01-19 15:16:27 -05:00
Michael Lieberman
72c9685450 Tweaks 2015-01-19 15:13:58 -05:00
Michael Lieberman
42afab5968 Partly move getEdges into EdgeTableWrapper 2015-01-19 15:12:52 -05:00
Michael Lieberman
332df1fea4 Clean up / reorganize non-useful / duplicate code 2015-01-19 15:06:37 -05:00
Michael Lieberman
6196dc7d8f Move key/value getVertices into VertexTableWrapper 2015-01-19 14:23:20 -05:00
Michael Lieberman
98a24bd62f Abstracted entry parser for vertex, edge and property parsers 2015-01-05 19:43:55 -05:00
Michael Lieberman
c9b7473df0 Remove cruft 2015-01-05 19:33:59 -05:00
Michael Lieberman
886c394afb Hide debug message 2015-01-05 19:33:15 -05:00
Michael Lieberman
8b7f8b7e9d Add log4j.properties 2015-01-05 19:33:03 -05:00
Michael Lieberman
25eb8e9979 Move cache items to their own package 2015-01-05 19:30:36 -05:00
Michael Lieberman
33042acedc Fix header licenses 2015-01-05 19:28:38 -05:00
Michael Lieberman
2465f65482 Merge branch 'master' into table-wrappers 2015-01-05 19:24:09 -05:00
Michael Lieberman
899ed892fc Move getEdges to EdgeTableWrapper
Make getVertices and getEdges cache their results
Remove confusing AccumuloEdge constructors and other methods
Added some debugging method for the backing tables
Bug fixes in unit tests
2015-01-05 19:23:57 -05:00
Michael Lieberman
ad7d405339 Update to next snapshot version 2015-01-05 16:58:06 -05:00
Michael Lieberman
37b4c8adc9 Refactor conversion from entries to elements as ElementParser and other classes 2015-01-05 15:10:06 -05:00
Michael Lieberman
76b0007434 Move getVertices() into VertexTableWrapper
Other cleanup
2015-01-05 14:33:48 -05:00
Michael Lieberman
3a08194ba3 Merge branch 'property-updates' into table-wrappers
Conflicts:
	src/main/java/edu/jhuapl/tinkerpop/AccumuloElement.java
	src/main/java/edu/jhuapl/tinkerpop/AccumuloGraph.java
2015-01-05 14:32:19 -05:00
Michael Lieberman
65b8d43d42 Cruft cleanup 2014-12-31 12:57:04 -05:00
Michael Lieberman
8b8fc0c7e6 Moved removeProperty functionality into AccumuloElement 2014-12-31 12:03:43 -05:00
Michael Lieberman
fec6fdc522 Partly move setProperty functionality into AccumuloElement 2014-12-31 11:51:53 -05:00
Michael Lieberman
439dc7ae93 Clean up test framework configuration 2014-12-31 01:39:51 -05:00
Michael Lieberman
ac68e334a6 Use a single writer in the table wrappers 2014-12-31 01:35:28 -05:00
Michael Lieberman
f12e78d836 Property cache updates
Added property caching unit test
Bug fix in property caching
Cruft cleanup and deprecate AccumuloGraph methods
2014-12-31 01:31:29 -05:00
Michael Lieberman
2d7a0d4c22 Property updates
Moved removeProperty to AccumuloElement
Added property cache unit tests
Moved flushing to GlobalInstances
Cruft cleanup

NOTE: Unit tests failing due to what appears to be race condition
2014-12-30 19:21:25 -05:00
Michael Lieberman
06bb292b42 Move getProperty and getPropertyKeys inside AccumuloElement
Other cruft cleanup
2014-12-30 16:56:20 -05:00
Michael Lieberman
6b85e54628 Move getVertices() into VertexTableWrapper 2014-12-30 15:46:20 -05:00
Michael Lieberman
ed77c11b1c Collect element caches into single object 2014-12-30 15:42:10 -05:00
Michael Lieberman
a61fc8ba20 Refactor getEdges to use Vertex object 2014-12-30 15:10:21 -05:00
Michael Lieberman
6613bcd569 Added edge endpoint add/delete
Added operation to batch delete elements based on range
Refactored add/delete operations into final classes
Updated documentation
2014-12-30 15:02:44 -05:00
Michael Lieberman
80ddbd3127 Initial commits for effort to genericize mutators based on graph operations 2014-12-30 14:38:15 -05:00
Michael Lieberman
c1429b3bf3 Several changes
Refactoring of wrappers to use GlobalInstances object
Added VertexTableWrapper and EdgeTableWrapper to GlobalInstances
Started moving vertex.getEdges() to VertexTableWrapper
2014-12-29 18:21:38 -05:00
Michael Lieberman
aa860b3441 Refactor property operations to use GlobalInstances object 2014-12-29 17:39:18 -05:00
Michael Lieberman
1c27add8e5 Refactor VertexTableWrapper and EdgeTableWrapper operations to use element objects 2014-12-29 17:31:30 -05:00
Michael Lieberman
9c2e07c395 Pass around a "GlobalInstances" object to simplify passing globals relevant to multiple AccumuloGraph objects 2014-12-29 17:26:53 -05:00
Michael Lieberman
bcc4af3599 Rename configuration methods for more clarity 2014-12-29 16:22:26 -05:00
Michael Lieberman
85544dea77 Merge branch 'master' into table-wrappers 2014-12-29 15:52:12 -05:00
Michael Lieberman
2ffdb81fe3 Refactoring property parsing to separate method in anticipation of vertex/edge-specific entry parsing 2014-12-24 16:52:18 -05:00
Michael Lieberman
c29983966b Add cacheAllProperties utility method 2014-12-24 16:38:59 -05:00
Michael Lieberman
bfba0ed665 Use ElementTableWrapper functionality for preloading edge properties 2014-12-24 16:08:15 -05:00
Michael Lieberman
a3181eac62 Move vertex property loading to ElementTableWrapper 2014-12-24 16:02:00 -05:00
Michael Lieberman
7d2d9bb45a Merge branch 'master' into table-wrappers 2014-12-24 15:26:02 -05:00
Michael Lieberman
fc09015445 Merge branch 'master' into table-wrappers 2014-12-24 13:30:02 -05:00
Michael Lieberman
cae1124562 Merge branch 'cache-updates' into table-wrappers
Conflicts:
	src/main/java/edu/jhuapl/tinkerpop/AccumuloElement.java
	src/main/java/edu/jhuapl/tinkerpop/AccumuloGraph.java
2014-12-24 13:07:30 -05:00
Michael Lieberman
aa4897ca27 Merge branch 'master' into table-wrappers
Conflicts:
	src/main/java/edu/jhuapl/tinkerpop/AccumuloGraph.java
2014-12-24 11:43:26 -05:00
Michael Lieberman
5affcea6e7 Merge branch 'master' into table-wrappers 2014-12-23 19:15:51 -05:00
Michael Lieberman
33cc53c00e Move edge writing methods to EdgeTableWrapper / VertexTableWrapper
Some more refactoring and cleanup
2014-12-23 19:01:00 -05:00
Michael Lieberman
0b8ab3361a Move vertex writing to VertexTableWrapper 2014-12-23 18:32:56 -05:00
Michael Lieberman
b991d84a28 Remove unnecessary parent argument to ScannerIterable 2014-12-23 17:53:06 -05:00
Michael Lieberman
bd7cd409d2 Simplification / abstraction of table wrappers 2014-12-23 17:32:44 -05:00
Michael Lieberman
b7d9d3f340 Initial commits of index wrappers
Remove unused methods from AccumuloGraphConfiguration
2014-12-23 17:27:38 -05:00
Michael Lieberman
0bf11f444e Create BaseTableWrapper and refactor ElementTableWrapper
Also added some Override annotations and other cleanup
2014-12-23 17:09:39 -05:00
Michael Lieberman
5d7442e024 Rename "tableoperations" to "tablewrapper" classes
Move TableWrapper classes to "tables" package
2014-12-23 16:53:45 -05:00
Michael Lieberman
670e4b5c85 Move setProperty functionality (not including indexes) to ElementTableOperations
Remove unused method in AccumuloGraph
Also, use AccumuloGraphException in more places
2014-12-23 16:42:07 -05:00
Michael Lieberman
2db2960b67 Move removeProperty functionality (not including indexes) to ElementTableOperations 2014-12-23 16:20:35 -05:00
Michael Lieberman
f378931c3a Move readPropertyKeys to ElementTableOperations 2014-12-23 15:53:05 -05:00
Michael Lieberman
b749943cc9 Move getProperty in AccumuloGraph to ElementTableOperations 2014-12-23 15:42:56 -05:00
Michael Lieberman
d20da99ad3 Initial commits for ElementTableOperations and subclasses 2014-12-23 15:39:16 -05:00
65 changed files with 4121 additions and 1687 deletions

View File

@@ -28,7 +28,7 @@ to Maven Central.
<dependency>
<groupId>edu.jhuapl.tinkerpop</groupId>
<artifactId>blueprints-accumulo-graph</artifactId>
<version>0.1.0</version>
<version>0.2.0</version>
</dependency>
```

6
changelog Normal file
View File

@@ -0,0 +1,6 @@
[2.1]
Change Log started
--109,103 Merged Metadata tables
--97 Added range queries
--95 Table structure documentation
--79 All flag for preloading

View File

@@ -3,7 +3,7 @@
<modelVersion>4.0.0</modelVersion>
<groupId>edu.jhuapl.tinkerpop</groupId>
<artifactId>blueprints-accumulo-graph</artifactId>
<version>0.1.0</version>
<version>0.2.2-SNAPSHOT</version>
<name>blueprints-accumulo-graph</name>
<description>An Accumulo-backed implementation of the Tinkerpop Blueprints graph API.</description>
<packaging>jar</packaging>

View File

@@ -32,6 +32,12 @@ import com.tinkerpop.blueprints.Edge;
import com.tinkerpop.blueprints.GraphFactory;
import com.tinkerpop.blueprints.Vertex;
import edu.jhuapl.tinkerpop.mutator.Mutators;
import edu.jhuapl.tinkerpop.mutator.edge.EdgeEndpointsMutator;
import edu.jhuapl.tinkerpop.mutator.edge.EdgeMutator;
import edu.jhuapl.tinkerpop.mutator.property.WritePropertyMutator;
import edu.jhuapl.tinkerpop.mutator.vertex.AddVertexMutator;
/**
* This class provides high-speed ingest into an {@link AccumuloGraph} instance
@@ -91,8 +97,8 @@ public final class AccumuloBulkIngester {
AccumuloGraphUtils.handleCreateAndClear(config);
mtbw = connector.createMultiTableBatchWriter(config.getBatchWriterConfig());
vertexWriter = mtbw.getBatchWriter(config.getVertexTable());
edgeWriter = mtbw.getBatchWriter(config.getEdgeTable());
vertexWriter = mtbw.getBatchWriter(config.getVertexTableName());
edgeWriter = mtbw.getBatchWriter(config.getEdgeTableName());
}
/**
@@ -108,9 +114,7 @@ public final class AccumuloBulkIngester {
* @throws MutationsRejectedException
*/
public PropertyBuilder addVertex(String id) throws MutationsRejectedException {
Mutation m = new Mutation(id);
m.put(AccumuloGraph.LABEL, AccumuloGraph.EXISTS, AccumuloGraph.EMPTY);
vertexWriter.addMutation(m);
Mutators.apply(vertexWriter, new AddVertexMutator(id));
return new PropertyBuilder(vertexWriter, id);
}
@@ -148,8 +152,7 @@ public final class AccumuloBulkIngester {
* @throws MutationsRejectedException
*/
public PropertyBuilder addEdge(String src, String dest, String label) throws MutationsRejectedException {
String eid = UUID.randomUUID().toString();
return addEdge(eid, src, dest, label);
return addEdge(UUID.randomUUID().toString(), src, dest, label);
}
/**
@@ -168,16 +171,8 @@ public final class AccumuloBulkIngester {
* @throws MutationsRejectedException
*/
public PropertyBuilder addEdge(String id, String src, String dest, String label) throws MutationsRejectedException {
Mutation m = new Mutation(id);
m.put(AccumuloGraph.LABEL, (dest + "_" + src).getBytes(), AccumuloByteSerializer.serialize(label));
edgeWriter.addMutation(m);
m = new Mutation(dest);
m.put(AccumuloGraph.INEDGE, (src + AccumuloGraph.IDDELIM + id).getBytes(), (AccumuloGraph.IDDELIM + label).getBytes());
vertexWriter.addMutation(m);
m = new Mutation(src);
m.put(AccumuloGraph.OUTEDGE, (dest + AccumuloGraph.IDDELIM + id).getBytes(), (AccumuloGraph.IDDELIM + label).getBytes());
vertexWriter.addMutation(m);
Mutators.apply(edgeWriter, new EdgeMutator.Add(id, src, dest, label));
Mutators.apply(vertexWriter, new EdgeEndpointsMutator.Add(id, src, dest, label));
return new PropertyBuilder(edgeWriter, id);
}
@@ -210,10 +205,7 @@ public final class AccumuloBulkIngester {
* @throws MutationsRejectedException
*/
private void addProperty(BatchWriter writer, String id, String key, Object value) throws MutationsRejectedException {
byte[] newByteVal = AccumuloByteSerializer.serialize(value);
Mutation m = new Mutation(id);
m.put(key.getBytes(), AccumuloGraph.EMPTY, newByteVal);
writer.addMutation(m);
Mutators.apply(writer, new WritePropertyMutator(id, key, value));
}
/**
@@ -277,14 +269,14 @@ public final class AccumuloBulkIngester {
* builder.finish();
* </PRE>
*/
public final class PropertyBuilder {
public static final class PropertyBuilder {
Mutation mutation;
BatchWriter writer;
final String id;
final BatchWriter writer;
PropertyBuilder(BatchWriter writer, String id) {
this.writer = writer;
this.mutation = new Mutation(id);
this.id = id;
}
/**
@@ -296,7 +288,13 @@ public final class AccumuloBulkIngester {
* @return
*/
public PropertyBuilder add(String key, Object value) {
mutation.put(key.getBytes(), AccumuloGraph.EMPTY, AccumuloByteSerializer.serialize(value));
for (Mutation m : new WritePropertyMutator(id, key, value).create()) {
try {
writer.addMutation(m);
} catch (MutationsRejectedException e) {
throw new AccumuloGraphException(e);
}
}
return this;
}
@@ -306,9 +304,7 @@ public final class AccumuloBulkIngester {
* @throws MutationsRejectedException
*/
public void finish() throws MutationsRejectedException {
if (mutation.size() > 0) {
writer.addMutation(mutation);
}
// No-op since Mutations are now added on the fly.
}
/**
@@ -317,7 +313,7 @@ public final class AccumuloBulkIngester {
* @return
*/
public String getId() {
return new String(mutation.getRow());
return id;
}
}
}

View File

@@ -26,21 +26,21 @@ import javax.xml.namespace.QName;
public final class AccumuloByteSerializer {
static final int NULL = 'n';
public static final int NULL = 'n';
static final int BYTE = 'b';
static final int SHORT = 's';
static final int CHARACTER = 'c';
static final int INTEGER = 'i';
static final int LONG = 'l';
static final int FLOAT = 'f';
static final int DOUBLE = 'd';
static final int BOOLEAN = 'o';
static final int DATE = 't';
static final int ENUM = 'e';
static final int STRING = 'a';
static final int SERIALIZABLE = 'x';
static final int QNAME = 'q';
public static final int BYTE = 'b';
public static final int SHORT = 's';
public static final int CHARACTER = 'c';
public static final int INTEGER = 'i';
public static final int LONG = 'l';
public static final int FLOAT = 'f';
public static final int DOUBLE = 'd';
public static final int BOOLEAN = 'o';
public static final int DATE = 't';
public static final int ENUM = 'e';
public static final int STRING = 'a';
public static final int SERIALIZABLE = 'x';
public static final int QNAME = 'q';
private AccumuloByteSerializer() {
@@ -53,6 +53,7 @@ public final class AccumuloByteSerializer {
}
};
@SuppressWarnings("unchecked")
public static <T> T deserialize(byte[] target) {
if (target[0] == NULL) {
return null;
@@ -92,6 +93,7 @@ public final class AccumuloByteSerializer {
case ENUM:
try {
String[] s = new String(target, 1, target.length - 1).split(":");
@SuppressWarnings("rawtypes")
Class<? extends Enum> clz = (Class<? extends Enum>) Class.forName(s[0]);
return (T) Enum.valueOf(clz, s[1]);
} catch (ClassNotFoundException cnfe) {

View File

@@ -14,70 +14,52 @@
*/
package edu.jhuapl.tinkerpop;
import java.util.Map;
import java.util.Map.Entry;
import org.apache.log4j.Logger;
import com.tinkerpop.blueprints.Direction;
import com.tinkerpop.blueprints.Edge;
import com.tinkerpop.blueprints.Vertex;
import com.tinkerpop.blueprints.util.ExceptionFactory;
import com.tinkerpop.blueprints.util.StringFactory;
/**
* TODO
*/
public class AccumuloEdge extends AccumuloElement implements Edge {
String label;
String inId;
String outId;
Vertex inVertex;
Vertex outVertex;
private static final Logger log = Logger.getLogger(AccumuloEdge.class);
AccumuloEdge(AccumuloGraph parent, String id) {
this(parent, id, null);
private String label;
private Vertex inVertex;
private Vertex outVertex;
public AccumuloEdge(GlobalInstances globals, String id) {
this(globals, id, null, null, null);
}
AccumuloEdge(AccumuloGraph parent, String id, String label) {
this(parent, id, label, (Vertex) null, (Vertex) null);
}
AccumuloEdge(AccumuloGraph parent, String id, String label, Vertex inVertex, Vertex outVertex) {
super(parent, id, Edge.class);
public AccumuloEdge(GlobalInstances globals, String id,
Vertex inVertex, Vertex outVertex, String label) {
super(globals, id, Edge.class);
this.label = label;
this.inVertex = inVertex;
this.outVertex = outVertex;
}
AccumuloEdge(AccumuloGraph parent, String id, String label, String inVertex, String outVertex) {
super(parent, id, Edge.class);
this.label = label;
this.inId = inVertex;
this.outId = outVertex;
}
@Override
public Vertex getVertex(Direction direction) throws IllegalArgumentException {
switch (direction) {
case IN:
if (inVertex == null) {
if (inId == null) {
inVertex = parent.getEdgeVertex(id, direction);
inId = inVertex.getId().toString();
} else {
inVertex = parent.getVertex(inId);
}
}
return inVertex;
case OUT:
if (outVertex == null) {
if (outId == null) {
outVertex = parent.getEdgeVertex(id, direction);
outId = outVertex.getId().toString();
} else {
outVertex = parent.getVertex(outId);
}
}
return outVertex;
case BOTH:
throw ExceptionFactory.bothIsNotSupported();
default:
throw new RuntimeException("Unexpected direction: " + direction);
if (!Direction.IN.equals(direction) && !Direction.OUT.equals(direction)) {
throw new IllegalArgumentException("Invalid direction: "+direction);
}
// The vertex information needs to be loaded.
if (inVertex == null || outVertex == null || label == null) {
log.debug("Loading information for edge: "+this);
globals.getEdgeWrapper().loadEndpointsAndLabel(this);
}
return Direction.IN.equals(direction) ? inVertex : outVertex;
}
@Override
@@ -91,31 +73,45 @@ public class AccumuloEdge extends AccumuloElement implements Edge {
@Override
public void remove() {
parent.removeEdge(this);
// Remove from named indexes.
super.removeElementFromNamedIndexes();
// If edge was removed already, forget it.
// This may happen due to self-loops...
if (!globals.getEdgeWrapper().elementExists(id)) {
return;
}
// Remove properties from key/value indexes.
Map<String, Object> props = globals.getEdgeWrapper()
.readAllProperties(this);
for (Entry<String,Object> ents : props.entrySet()) {
globals.getEdgeKeyIndexWrapper().removePropertyFromIndex(this,
ents.getKey(), ents.getValue());
}
// Get rid of the endpoints and edge themselves.
globals.getVertexWrapper().deleteEdgeEndpoints(this);
globals.getEdgeWrapper().deleteEdge(this);
// Remove element from cache.
globals.getCaches().remove(id, Edge.class);
globals.checkedFlush();
}
public String getInId() {
return inId;
public void setVertices(AccumuloVertex inVertex, AccumuloVertex outVertex) {
this.inVertex = inVertex;
this.outVertex = outVertex;
}
public String getOutId() {
return outId;
}
protected void setInId(String id){
inId = id;
}
protected void setOutId(String id){
outId = id;
}
protected void setLabel(String label){
public void setLabel(String label) {
this.label = label;
}
@Override
public String toString() {
return "[" + getId() + ":" + getVertex(Direction.OUT) + " -> " + getLabel() + " -> " + getVertex(Direction.IN) + "]";
return "[" + getId() + ":" + inVertex + " -> " + label + " -> " + outVertex + "]";
}
}

View File

@@ -16,71 +16,153 @@ package edu.jhuapl.tinkerpop;
import java.util.Set;
import org.apache.accumulo.core.util.Pair;
import com.tinkerpop.blueprints.Element;
import com.tinkerpop.blueprints.Index;
import com.tinkerpop.blueprints.util.StringFactory;
import edu.jhuapl.tinkerpop.cache.PropertyCache;
/**
* TODO
*/
public abstract class AccumuloElement implements Element {
protected AccumuloGraph parent;
protected GlobalInstances globals;
protected String id;
private Class<? extends Element> type;
private PropertyCache propertyCache;
protected AccumuloElement(AccumuloGraph parent, String id, Class<? extends Element> type) {
this.parent = parent;
protected AccumuloElement(GlobalInstances globals,
String id, Class<? extends Element> type) {
this.globals = globals;
this.id = id;
this.type = type;
}
@Override
public <T> T getProperty(String key) {
/**
* Create properties cache if it doesn't exist,
* and preload any properties.
*/
private void makeCache() {
if (propertyCache == null) {
// Lazily create the properties cache.
// We will create it here just in case the parent does not actually
// pre-load any data. Note it also may be created in the
// cacheProperty method, as well, in the event a class pre-loads
// data before a call is made to obtain it.
propertyCache = new PropertyCache(parent.config);
propertyCache = new PropertyCache(globals.getConfig());
parent.preloadProperties(this, type);
}
T val = propertyCache.get(key);
if (val != null) {
return val;
} else {
Pair<Integer, T> pair = parent.getProperty(type, id, key);
if (pair.getFirst() != null) {
cacheProperty(key, pair.getSecond());
// Preload any keys, if needed.
String[] preloadKeys = globals.getConfig().getPreloadedProperties();
if (preloadKeys != null) {
propertyCache.putAll(globals.getElementWrapper(type)
.readProperties(this, preloadKeys));
}
return pair.getSecond();
}
}
@Override
public <T> T getProperty(String key) {
makeCache();
// Get from property cache.
T value = propertyCache.get(key);
// If not cached, get it from the backing table.
if (value == null) {
value = globals.getElementWrapper(type).readProperty(this, key);
}
// Cache the new value.
if (value != null) {
propertyCache.put(key, value);
}
return value;
}
@Override
public Set<String> getPropertyKeys() {
return parent.getPropertyKeys(type, id);
return globals.getElementWrapper(type).readPropertyKeys(this);
}
@Override
public void setProperty(String key, Object value) {
parent.setProperty(type, id, key, value);
cacheProperty(key, value);
makeCache();
globals.getKeyIndexTableWrapper(type).setPropertyForIndex(this, key, value);
// MDL 31 Dec 2014: The above calls getProperty, so this
// order is important (for now).
globals.getElementWrapper(type).writeProperty(this, key, value);
globals.checkedFlush();
setPropertyInMemory(key, value);
}
/**
* Set a property but only in the instantiated object,
* not in the backing store.
* @param key
* @param value
*/
public void setPropertyInMemory(String key, Object value) {
makeCache();
propertyCache.put(key, value);
}
@Override
public <T> T removeProperty(String key) {
if (propertyCache != null) {
// we have the cached value but we still need to pass this on to the
// parent so it can actually remove the data from the backing store.
// Since we have to do that anyway, we will use the parent's value
// instead of the cache value to to be as up-to-date as possible.
// Of course we still need to clear out the cached value...
propertyCache.remove(key);
if (StringFactory.LABEL.equals(key) ||
Constants.LABEL.equals(key)) {
throw new AccumuloGraphException("Cannot remove the " + StringFactory.LABEL + " property.");
}
return parent.removeProperty(type, id, key);
makeCache();
T value = getProperty(key);
if (value != null) {
globals.getElementWrapper(type).clearProperty(this, key);
globals.checkedFlush();
}
globals.getKeyIndexTableWrapper(type).removePropertyFromIndex(this, key, value);
// MDL 31 Dec 2014: AccumuloGraph.removeProperty
// calls getProperty which populates the cache.
// So the order here is important (for now).
removePropertyInMemory(key);
return value;
}
/**
* Remove element from all named indexes.
* @param element
*/
protected void removeElementFromNamedIndexes() {
for (Index<? extends Element> index : globals.getIndexMetadataWrapper().getIndices()) {
((AccumuloIndex<? extends Element>) index).getWrapper().removeElementFromIndex(this);
}
}
/**
* Remove a property but only in the instantiated
* object, not the backing store.
* @param key
*/
public void removePropertyInMemory(String key) {
makeCache();
propertyCache.remove(key);
}
/**
* Return the properties currently cached in memory.
* @return
*/
public Iterable<String> getPropertyKeysInMemory() {
makeCache();
return propertyCache.keySet();
}
/**
* Retrieve a property from memory.
* @param key
* @return
*/
public Object getPropertyInMemory(String key) {
makeCache();
return propertyCache.get(key);
}
@Override
@@ -97,19 +179,20 @@ public abstract class AccumuloElement implements Element {
} else if (!obj.getClass().equals(getClass())) {
return false;
} else {
return this.id.equals(((AccumuloElement) obj).id);
return id.equals(((AccumuloElement) obj).id);
}
}
@Override
public int hashCode() {
return getClass().hashCode() ^ id.hashCode();
}
void cacheProperty(String key, Object value) {
if (propertyCache == null) {
propertyCache = new PropertyCache(parent.config);
}
propertyCache.put(key, value);
/**
* Internal method for unit tests.
* @return
*/
PropertyCache getPropertyCache() {
return propertyCache;
}
}

File diff suppressed because it is too large Load Diff

View File

@@ -18,6 +18,9 @@ import java.io.File;
import java.io.IOException;
import java.io.Serializable;
import java.lang.reflect.Field;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.attribute.FileAttribute;
import java.util.Arrays;
import java.util.Iterator;
import java.util.List;
@@ -41,8 +44,11 @@ import org.apache.accumulo.minicluster.MiniAccumuloCluster;
import org.apache.commons.configuration.AbstractConfiguration;
import org.apache.commons.configuration.Configuration;
import org.apache.commons.configuration.PropertiesConfiguration;
import org.apache.commons.configuration.event.ConfigurationEvent;
import org.apache.commons.configuration.event.ConfigurationListener;
import org.apache.hadoop.io.Text;
import com.tinkerpop.blueprints.Graph;
import com.tinkerpop.blueprints.IndexableGraph;
import com.tinkerpop.blueprints.KeyIndexableGraph;
@@ -60,15 +66,26 @@ implements Serializable {
/**
* Backing configuration object.
*/
private Configuration conf;
private PropertiesConfiguration conf;
/**
* Maintain a connector instance that will be
* reused for {@link #getConnector()}.
*/
private Connector connector;
/**
* Temp directory used by getInstance when a Mini InstanceType is used.
*/
private String miniClusterTempDir;
private File miniClusterTempDir;
private MiniAccumuloCluster accumuloMiniCluster;
/**
* The {@link AccumuloGraph} class.
*/
public static final Class<? extends Graph> ACCUMULO_GRAPH_CLASS = AccumuloGraph.class;
/**
* The fully-qualified class name of the class that implements
* the TinkerPop Graph interface.
@@ -76,7 +93,7 @@ implements Serializable {
* which type to instantiate.
*/
public static final String ACCUMULO_GRAPH_CLASSNAME =
AccumuloGraph.class.getCanonicalName();
ACCUMULO_GRAPH_CLASS.getCanonicalName();
/**
* An enumeration used by {@link AccumuloGraphConfiguration#setInstanceType(InstanceType)}
@@ -112,7 +129,8 @@ implements Serializable {
public static final String SPLITS = "blueprints.accumulo.splits";
public static final String COLVIS = "blueprints.accumulo.columnVisibility";
public static final String SKIP_CHECKS = "blueprints.accumulo.skipExistenceChecks";
public static final String PRELOAD_PROPERTIES = "blueprints.accumulo.property.preload";
public static final String PRELOADED_PROPERTIES = "blueprints.accumulo.property.preload";
public static final String PRELOAD_ALL_PROPERTIES = "blueprints.accumulo.property.preload.all";
public static final String PROPERTY_CACHE_TIMEOUT = "blueprints.accumulo.propertyCacheTimeout";
public static final String EDGE_CACHE_SIZE = "blueprints.accumulo.edgeCacheSize";
public static final String EDGE_CACHE_TIMEOUT = "blueprints.accumulo.edgeCacheTimeout";
@@ -170,6 +188,7 @@ implements Serializable {
setInstanceType(InstanceType.Distributed);
setAuthorizations(Constants.NO_AUTHS);
setSkipExistenceChecks(false);
setPreloadAllProperties(false);
}
/**
@@ -180,6 +199,15 @@ implements Serializable {
return conf;
}
/**
* Create a copy of this configuration.
* This is needed to change config parameters
* if {@link #getConnector()} has been used.
*/
public AccumuloGraphConfiguration clone() {
return new AccumuloGraphConfiguration(this);
}
public boolean getCreate() {
return conf.getBoolean(Keys.CREATE);
}
@@ -228,6 +256,7 @@ implements Serializable {
InstanceType.Mini.equals(type)) {
setUser("root");
setPassword("");
setInstanceName("");
setCreate(true);
}
@@ -708,11 +737,26 @@ implements Serializable {
return name.matches("^[A-Za-z0-9_]+$");
}
public boolean getPreloadAllProperties() {
return conf.getBoolean(Keys.PRELOAD_ALL_PROPERTIES);
}
/**
* If true, retrieve all properties for elements when
* retrieving them from Accumulo. This can be used
* in lieu of {@link #setPreloadedProperties(String[])}
* when all properties are needed. Defaults to false.
* @param preload
* @return
*/
public AccumuloGraphConfiguration setPreloadAllProperties(boolean preload) {
conf.setProperty(Keys.PRELOAD_ALL_PROPERTIES, preload);
return this;
}
public String[] getPreloadedProperties() {
if (conf.containsKey(Keys.PRELOAD_PROPERTIES)) {
return conf.getStringArray(Keys.PRELOAD_PROPERTIES);
}
return null;
return conf.containsKey(Keys.PRELOADED_PROPERTIES) ?
conf.getStringArray(Keys.PRELOADED_PROPERTIES) : null;
}
/**
@@ -742,15 +786,13 @@ implements Serializable {
throw new NullPointerException("Property keys cannot be null.");
}
conf.setProperty(Keys.PRELOAD_PROPERTIES, propertyKeys);
conf.setProperty(Keys.PRELOADED_PROPERTIES, propertyKeys);
return this;
}
public String[] getPreloadedEdgeLabels() {
if (conf.containsKey(Keys.PRELOAD_EDGES)) {
return conf.getStringArray(Keys.PRELOAD_EDGES);
}
return null;
return conf.containsKey(Keys.PRELOAD_EDGES) ?
conf.getStringArray(Keys.PRELOAD_EDGES) : null;
}
/**
@@ -844,6 +886,7 @@ implements Serializable {
/**
* Create a {@link Connector} from this configuration.
* <p/>Note: Once this is called, the configuration may not be modified.
* @return
* @throws AccumuloException
* @throws AccumuloSecurityException
@@ -852,43 +895,53 @@ implements Serializable {
*/
public Connector getConnector() throws AccumuloException, AccumuloSecurityException,
IOException, InterruptedException {
Instance inst = null;
switch (getInstanceType()) {
case Distributed:
if (getInstanceName() == null) {
throw new IllegalArgumentException("Must specify instance name for distributed mode");
} else if (getZooKeeperHosts() == null) {
throw new IllegalArgumentException("Must specify ZooKeeper hosts for distributed mode");
if (connector == null) {
Instance inst = null;
switch (getInstanceType()) {
case Mini:
if (miniClusterTempDir == null) {
miniClusterTempDir = Files.createTempDirectory("accumulo-mini").toFile();
}
accumuloMiniCluster = new MiniAccumuloCluster(miniClusterTempDir, ""); // conf.getString(PASSWORD)
try {
accumuloMiniCluster.start();
} catch (Exception ex) {
throw new AccumuloGraphException(ex);
}
this.setInstanceName(accumuloMiniCluster.getInstanceName());
this.setZooKeeperHosts(accumuloMiniCluster.getZooKeepers());
this.setPassword("");
case Distributed:
if (getInstanceName() == null) {
throw new IllegalArgumentException("Must specify instance name for distributed mode");
} else if (getZooKeeperHosts() == null) {
throw new IllegalArgumentException("Must specify ZooKeeper hosts for distributed mode");
}
inst = new ZooKeeperInstance(getInstanceName(), getZooKeeperHosts());
break;
case Mock:
inst = new MockInstance(getInstanceName());
break;
default:
throw new AccumuloGraphException("Unexpected instance type: " + inst);
}
connector = inst.getConnector(getUser(), new PasswordToken(getPassword()));
// Make the configuration immutable.
conf.addConfigurationListener(new ConfigurationListener() {
@Override
public void configurationChanged(ConfigurationEvent event) {
throw new AccumuloGraphException("You may not modify the configuration after calling getConnector()");
}
inst = new ZooKeeperInstance(getInstanceName(), getZooKeeperHosts());
break;
case Mini:
File dir = null;
if (miniClusterTempDir == null) {
dir = createTempDir();
dir.deleteOnExit();
} else {
// already set by setMiniClusterTempDir(), It should be cleaned up outside of this class.
dir = new File(miniClusterTempDir);
}
accumuloMiniCluster = new MiniAccumuloCluster(dir, ""); // conf.getString(PASSWORD)
try {
accumuloMiniCluster.start();
} catch (Exception ex) {
ex.printStackTrace();
System.out.println("");
}
inst = new ZooKeeperInstance(accumuloMiniCluster.getInstanceName(), accumuloMiniCluster.getZooKeepers());
throw new UnsupportedOperationException("TODO");
case Mock:
inst = new MockInstance(getInstanceName());
break;
default:
throw new RuntimeException("Unexpected instance type: " + inst);
});
}
Connector c = inst.getConnector(getUser(), new PasswordToken(getPassword()));
return c;
return connector;
}
/**
@@ -918,46 +971,66 @@ implements Serializable {
* @param miniClusterTempDir
*/
public AccumuloGraphConfiguration setMiniClusterTempDir(String miniClusterTempDir) {
this.miniClusterTempDir = miniClusterTempDir;
this.miniClusterTempDir = new File(miniClusterTempDir);
return this;
}
public String getVertexTable() {
/**
* Name of vertex table (keyed by vertex id).
* @return
*/
public String getVertexTableName() {
return getGraphName() + "_vertex";
}
public String getEdgeTable() {
/**
* Name of edge table (keyed by edge id).
* @return
*/
public String getEdgeTableName() {
return getGraphName() + "_edge";
}
public String getKeyMetadataTable() {
return getMetadataTable() + "KEY";
/**
* Name of vertex key index table (keyed on
* vertex property keys).
* @return
*/
public String getVertexKeyIndexTableName() {
return getGraphName() + "_vertex_key_index";
}
String getVertexIndexTable() {
return getGraphName() + "_vertex_index";
/**
* Name of edge key index table (keyed on
* edge property keys).
* @return
*/
public String getEdgeKeyIndexTableName() {
return getGraphName() + "_edge_key_index";
}
String getEdgeIndexTable() {
return getGraphName() + "_edge_index";
/**
* Table of the index with given name (keyed
* on property keys of the given element type).
* @param indexName
* @return
*/
public String getNamedIndexTableName(String indexName) {
return getGraphName() + "_index_" + indexName;
}
String getMetadataTable() {
return getGraphName() + "_meta";
}
String getKeyVertexIndexTable() {
return getGraphName() + "_vertex_index_key";
}
String getKeyEdgeIndexTable() {
return getGraphName() + "_edge_index_key";
/**
* Table containing index metadata.
* @return
*/
public String getIndexMetadataTableName() {
return getGraphName() + "_index_metadata";
}
List<String> getTableNames() {
return Arrays.asList(getVertexTable(),
getEdgeTable(), getVertexIndexTable(), getEdgeIndexTable(),
getMetadataTable(), getKeyMetadataTable());
return Arrays.asList(getVertexTableName(),
getEdgeTableName(), getVertexKeyIndexTableName(), getEdgeKeyIndexTableName(),
getIndexMetadataTableName());
}
/**
@@ -973,20 +1046,20 @@ implements Serializable {
checkPropertyValue(Keys.PASSWORD, getPassword(), false);
// no break intentional
case Mini:
checkPropertyValue(Keys.INSTANCE, getInstanceName(), false);
checkPropertyValue(Keys.PASSWORD, getPassword(), true);
// checkPropertyValue(Keys.INSTANCE, getInstanceName(), false);
// checkPropertyValue(Keys.PASSWORD, getPassword(), true);
// no break intentional
case Mock:
checkPropertyValue(Keys.GRAPH_NAME, getGraphName(), false);
break;
default:
throw new RuntimeException("Unexpected instance type: " + getInstanceType());
throw new AccumuloGraphException("Unexpected instance type: " + getInstanceType());
}
int timeout = getPropertyCacheTimeout(null);
if (timeout <= 0) {
String[] props = conf.getStringArray(Keys.PRELOAD_PROPERTIES);
String[] props = conf.getStringArray(Keys.PRELOADED_PROPERTIES);
for (int i = 0; i < props.length; i++) {
timeout = getPropertyCacheTimeout(props[i]);
if (timeout <= 0) {
@@ -995,9 +1068,15 @@ implements Serializable {
}
}
if (timeout <= 0 && conf.getProperty(Keys.PRELOAD_PROPERTIES) != null) {
if (getPreloadAllProperties() && getPreloadedProperties() != null) {
throw new IllegalArgumentException("Cannot preload all properties"
+ " and specified properties simultaneously");
}
if (timeout <= 0 && (getPreloadedProperties() != null || getPreloadAllProperties())) {
throw new IllegalArgumentException("You cannot preload properties "
+ "without first setting #propertyCacheTimeout(String property, int millis) " + "to a positive value.");
+ "without first setting #propertyCacheTimeout(String property, int millis) "
+ "to a positive value.");
}
}
@@ -1010,12 +1089,6 @@ implements Serializable {
}
}
private File createTempDir() throws IOException {
File temp = File.createTempFile(AccumuloGraphConfiguration.class.getSimpleName(), ".mini.tmp");
temp.delete();
temp.mkdir();
return temp;
}
/**
* Print out this configuration.
@@ -1043,7 +1116,7 @@ implements Serializable {
try {
keys.add((String) field.get(null));
} catch (Exception e) {
throw new RuntimeException(e);
throw new AccumuloGraphException(e);
}
}
@@ -1057,6 +1130,7 @@ implements Serializable {
* @param zookeeperHosts
* @return
*/
@Deprecated
public AccumuloGraphConfiguration setZookeeperHosts(String zookeeperHosts) {
return setZooKeeperHosts(zookeeperHosts);
}
@@ -1066,6 +1140,7 @@ implements Serializable {
* @param skip
* @return
*/
@Deprecated
public AccumuloGraphConfiguration skipExistenceChecks(boolean skip) {
return setSkipExistenceChecks(skip);
}
@@ -1074,6 +1149,7 @@ implements Serializable {
* @deprecated This is an old method name. Use {@link #getAutoIndex()}.
* @return
*/
@Deprecated
public boolean isAutoIndex() {
return getAutoIndex();
}
@@ -1082,6 +1158,7 @@ implements Serializable {
* @deprecated This is an old method name. Use {@link #getIndexableGraphDisabled()}.
* @return
*/
@Deprecated
public boolean isIndexableGraphDisabled() {
return getIndexableGraphDisabled();
}
@@ -1090,6 +1167,7 @@ implements Serializable {
* @deprecated This is an old method name. Use {@link #getCreate()}.
* @return
*/
@Deprecated
public boolean isCreate() {
return getCreate();
}
@@ -1098,6 +1176,7 @@ implements Serializable {
* @deprecated This is an old method name. Use {@link #getClear()}.
* @return
*/
@Deprecated
public boolean isClear() {
return getClear();
}
@@ -1106,6 +1185,7 @@ implements Serializable {
* @deprecated This is an old method name. Use {@link #getInstanceName()}.
* @return
*/
@Deprecated
public String getInstance() {
return getInstanceName();
}
@@ -1114,6 +1194,7 @@ implements Serializable {
* @deprecated This is an old method name. Use {@link #getAutoFlush()}.
* @return
*/
@Deprecated
public boolean isAutoFlush() {
return getAutoFlush();
}
@@ -1122,6 +1203,7 @@ implements Serializable {
* @deprecated This is an old method name. Use {@link #getPropertyCacheTimeout(String)}.
* @return
*/
@Deprecated
public Integer getPropertyCacheTimeoutMillis(String property) {
return getPropertyCacheTimeout(property);
}
@@ -1130,6 +1212,7 @@ implements Serializable {
* @deprecated This is an old method name. Use {@link #getEdgeCacheTimeout()}.
* @return
*/
@Deprecated
public Integer getEdgeCacheTimeoutMillis() {
return getEdgeCacheTimeout();
}
@@ -1138,6 +1221,7 @@ implements Serializable {
* @deprecated This is an old method name. Use {@link #getVertexCacheTimeout()}.
* @return
*/
@Deprecated
public Integer getVertexCacheTimeoutMillis() {
return getVertexCacheTimeout();
}
@@ -1146,6 +1230,7 @@ implements Serializable {
* @deprecated This is an old method name. Use {@link #getSkipExistenceChecks()}.
* @return
*/
@Deprecated
public boolean skipExistenceChecks() {
return getSkipExistenceChecks();
}
@@ -1154,6 +1239,7 @@ implements Serializable {
* @deprecated This is an old method name. Use {@link #getGraphName()}.
* @return
*/
@Deprecated
public String getName() {
return getGraphName();
}
@@ -1162,6 +1248,7 @@ implements Serializable {
* @deprecated This is an old method name. Use {@link #getPreloadedEdgeLabels()}.
* @return
*/
@Deprecated
public String[] getPreloadedEdges() {
return getPreloadedEdgeLabels();
}

View File

@@ -15,11 +15,15 @@
package edu.jhuapl.tinkerpop;
import java.util.SortedSet;
import java.util.UUID;
import org.apache.accumulo.core.client.admin.TableOperations;
import org.apache.hadoop.io.Text;
final class AccumuloGraphUtils {
import com.tinkerpop.blueprints.util.ExceptionFactory;
import com.tinkerpop.blueprints.util.StringFactory;
public final class AccumuloGraphUtils {
/**
* Create and/or clear existing graph tables for the given configuration.
@@ -76,4 +80,45 @@ final class AccumuloGraphUtils {
throw new IllegalArgumentException(e);
}
}
/**
* Generate an element id.
* @return
*/
public static String generateId() {
return UUID.randomUUID().toString();
}
/**
* Ensure that the given key/value don't conflict with
* Blueprints reserved words.
* @param key
* @param value
*/
public static void validateProperty(String key, Object value) {
nullCheckProperty(key, value);
if (key.equals(StringFactory.ID)) {
throw ExceptionFactory.propertyKeyIdIsReserved();
} else if (key.equals(StringFactory.LABEL)) {
throw ExceptionFactory.propertyKeyLabelIsReservedForEdges();
} else if (value == null) {
throw ExceptionFactory.propertyValueCanNotBeNull();
}
}
/**
* Disallow null keys/values and throw appropriate
* Blueprints exceptions.
* @param key
* @param value
*/
public static void nullCheckProperty(String key, Object value) {
if (key == null) {
throw ExceptionFactory.propertyKeyCanNotBeNull();
} else if (value == null) {
throw ExceptionFactory.propertyValueCanNotBeNull();
} else if (key.trim().equals(StringFactory.EMPTY_STRING)) {
throw ExceptionFactory.propertyKeyCanNotBeEmpty();
}
}
}

View File

@@ -15,82 +15,76 @@
package edu.jhuapl.tinkerpop;
import java.util.Iterator;
import java.util.Map.Entry;
import org.apache.accumulo.core.client.BatchWriter;
import org.apache.accumulo.core.client.MutationsRejectedException;
import org.apache.accumulo.core.client.Scanner;
import org.apache.accumulo.core.client.ScannerBase;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Mutation;
import org.apache.accumulo.core.data.Range;
import org.apache.accumulo.core.data.Value;
import org.apache.accumulo.core.util.PeekingIterator;
import org.apache.hadoop.io.Text;
import com.tinkerpop.blueprints.CloseableIterable;
import com.tinkerpop.blueprints.Edge;
import com.tinkerpop.blueprints.Element;
import com.tinkerpop.blueprints.Index;
import com.tinkerpop.blueprints.IndexableGraph;
import edu.jhuapl.tinkerpop.tables.index.NamedIndexTableWrapper;
/**
* Accumulo-based implementation for {@link IndexableGraph}.
* @param <T>
*/
public class AccumuloIndex<T extends Element> implements Index<T> {
Class<T> indexedType;
AccumuloGraph parent;
String indexName;
String tableName;
private final GlobalInstances globals;
private final Class<T> indexedType;
private final String indexName;
private final NamedIndexTableWrapper indexWrapper;
public AccumuloIndex(Class<T> t, AccumuloGraph parent, String indexName) {
this.indexedType = t;
this.parent = parent;
public AccumuloIndex(GlobalInstances globals, String indexName, Class<T> indexedType) {
this.globals = globals;
this.indexName = indexName;
tableName = parent.config.getGraphName() + "_index_" + indexName;// + "_" +
// t;
this.indexedType = indexedType;
try {
if (!parent.config.getConnector().tableOperations().exists(tableName)) {
parent.config.getConnector().tableOperations().create(tableName);
if (!globals.getConfig().getConnector()
.tableOperations().exists(getTableName())) {
globals.getConfig().getConnector()
.tableOperations().create(getTableName());
}
} catch (Exception e) {
throw new RuntimeException(e);
throw new AccumuloGraphException(e);
}
indexWrapper = new NamedIndexTableWrapper(globals, indexedType, indexName);
}
@Override
public String getIndexName() {
return indexName;
}
public String getTableName() {
return globals.getConfig().getNamedIndexTableName(indexName);
}
public NamedIndexTableWrapper getWrapper() {
return indexWrapper;
}
@Override
public Class<T> getIndexClass() {
return indexedType;
}
@Override
public void put(String key, Object value, Element element) {
element.setProperty(key, value);
Mutation m = new Mutation(AccumuloByteSerializer.serialize(value));
m.put(key.getBytes(), element.getId().toString().getBytes(), "".getBytes());
BatchWriter w = getWriter();
try {
w.addMutation(m);
w.flush();
} catch (MutationsRejectedException e) {
e.printStackTrace();
}
indexWrapper.setPropertyForIndex(element, key, value, true);
}
@Override
public CloseableIterable<T> get(String key, Object value) {
Scanner scan = getScanner();
byte[] id = AccumuloByteSerializer.serialize(value);
scan.setRange(new Range(new Text(id), new Text(id)));
scan.fetchColumnFamily(new Text(key));
return new IndexIterable(parent, scan, indexedType);
return indexWrapper.readElementsFromIndex(key, value);
}
@Override
public CloseableIterable<T> query(String key, Object query) {
throw new UnsupportedOperationException();
}
@Override
public long count(String key, Object value) {
CloseableIterable<T> iterable = get(key, value);
Iterator<T> iter = iterable.iterator();
@@ -103,75 +97,8 @@ public class AccumuloIndex<T extends Element> implements Index<T> {
return count;
}
public void remove(String key, Object value, Element element) {
Mutation m = new Mutation(AccumuloByteSerializer.serialize(value));
m.putDelete(key.getBytes(), element.getId().toString().getBytes());
BatchWriter w = getWriter();
try {
w.addMutation(m);
w.flush();
} catch (MutationsRejectedException e) {
e.printStackTrace();
}
}
private BatchWriter getWriter() {
return parent.getWriter(tableName);
}
private Scanner getScanner() {
return parent.getScanner(tableName);
}
public class IndexIterable implements CloseableIterable<T> {
AccumuloGraph parent;
ScannerBase scan;
boolean isClosed;
Class<T> indexedType;
IndexIterable(AccumuloGraph parent, ScannerBase scan, Class<T> t) {
this.scan = scan;
this.parent = parent;
isClosed = false;
indexedType = t;
}
public Iterator<T> iterator() {
if (!isClosed) {
return new ScannerIterable<T>(parent, scan) {
@Override
public T next(PeekingIterator<Entry<Key, Value>> iterator) {
String id = iterator.next()
.getKey().getColumnQualifier().toString();
// TODO better use of information readily
// available...
if (indexedType.equals(Edge.class)) {
return (T) new AccumuloEdge(parent, id);
}
else {
return (T) new AccumuloVertex(parent, id);
}
}
}.iterator();
}
else {
return null;
}
}
public void close() {
if (!isClosed) {
scan.close();
isClosed = true;
}
}
}
@Override
public Class<T> getIndexClass() {
return indexedType;
public void remove(String key, Object value, Element element) {
indexWrapper.removePropertyFromIndex(element, key, value);
}
}

View File

@@ -14,26 +14,34 @@
*/
package edu.jhuapl.tinkerpop;
import java.util.Map;
import java.util.Map.Entry;
import com.tinkerpop.blueprints.CloseableIterable;
import com.tinkerpop.blueprints.Direction;
import com.tinkerpop.blueprints.Edge;
import com.tinkerpop.blueprints.Vertex;
import com.tinkerpop.blueprints.VertexQuery;
import com.tinkerpop.blueprints.util.DefaultVertexQuery;
import com.tinkerpop.blueprints.util.ExceptionFactory;
/**
* TODO
*/
public class AccumuloVertex extends AccumuloElement implements Vertex {
AccumuloVertex(AccumuloGraph parent, String id) {
super(parent, id, Vertex.class);
public AccumuloVertex(GlobalInstances globals, String id) {
super(globals, id, Vertex.class);
}
@Override
public Iterable<Edge> getEdges(Direction direction, String... labels) {
return parent.getEdges(id, direction, labels);
return globals.getVertexWrapper().getEdges(this, direction, labels);
}
@Override
public Iterable<Vertex> getVertices(Direction direction, String... labels) {
return parent.getVertices(id, direction, labels);
return globals.getVertexWrapper().getVertices(this, direction, labels);
}
@Override
@@ -43,12 +51,74 @@ public class AccumuloVertex extends AccumuloElement implements Vertex {
@Override
public Edge addEdge(String label, Vertex inVertex) {
return parent.addEdge(null, this, inVertex, label);
return addEdge(null, label, inVertex);
}
/**
* Add an edge as with {@link #addEdge(String, Vertex)},
* but with a specified edge id.
* @param id
* @param label
* @param inVertex
* @return
*/
public Edge addEdge(Object id, String label, Vertex inVertex) {
if (label == null) {
throw ExceptionFactory.edgeLabelCanNotBeNull();
}
if (id == null) {
id = AccumuloGraphUtils.generateId();
}
String myID = id.toString();
AccumuloEdge edge = new AccumuloEdge(globals, myID, inVertex, this, label);
// TODO we arent suppose to make sure the given edge ID doesn't already
// exist?
globals.getEdgeWrapper().writeEdge(edge);
globals.getVertexWrapper().writeEdgeEndpoints(edge);
globals.checkedFlush();
globals.getCaches().cache(edge, Edge.class);
return edge;
}
@Override
public void remove() {
parent.removeVertex(this);
globals.getCaches().remove(getId(), Vertex.class);
super.removeElementFromNamedIndexes();
// Throw exception if the element does not exist.
if (!globals.getVertexWrapper().elementExists(id)) {
throw ExceptionFactory.vertexWithIdDoesNotExist(getId());
}
// Remove properties from key/value indexes.
Map<String, Object> props = globals.getVertexWrapper()
.readAllProperties(this);
for (Entry<String,Object> ent : props.entrySet()) {
globals.getVertexKeyIndexWrapper().removePropertyFromIndex(this,
ent.getKey(), ent.getValue());
}
// Remove edges incident to this vertex.
CloseableIterable<Edge> iter = (CloseableIterable<Edge>)getEdges(Direction.BOTH);
for (Edge edge : iter) {
edge.remove();
}
iter.close();
globals.checkedFlush();
// Get rid of the vertex.
globals.getVertexWrapper().deleteVertex(this);
globals.checkedFlush();
}
@Override

View File

@@ -0,0 +1,46 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop;
/**
* Collect up various constants here.
* @author Michael Lieberman
*
*/
public class Constants {
private Constants() { }
/**
* Separate element ids in Accumulo entries.
*/
public static final String ID_DELIM = "__DELIM__";
public static final byte[] EMPTY = new byte[0];
/**
* Prefixes for various Accumulo entries.
*/
public static final String LABEL = "__LABEL__";
public static final String IN_EDGE = "__IN_EDGE__";
public static final String OUT_EDGE = "__OUT_EDGE__";
public static final String EXISTS = "__EXISTS__";
/**
* Type of metadata to distinguish between
* entries in the metadata table.
*/
public static enum IndexMetadataEntryType {__INDEX_KEY__, __INDEX_NAME__};
}

View File

@@ -0,0 +1,114 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop;
import org.apache.accumulo.core.client.MultiTableBatchWriter;
import org.apache.accumulo.core.client.MutationsRejectedException;
import com.tinkerpop.blueprints.Edge;
import com.tinkerpop.blueprints.Element;
import com.tinkerpop.blueprints.Vertex;
import edu.jhuapl.tinkerpop.cache.ElementCaches;
import edu.jhuapl.tinkerpop.tables.core.EdgeTableWrapper;
import edu.jhuapl.tinkerpop.tables.core.ElementTableWrapper;
import edu.jhuapl.tinkerpop.tables.core.VertexTableWrapper;
import edu.jhuapl.tinkerpop.tables.index.BaseKeyIndexTableWrapper;
import edu.jhuapl.tinkerpop.tables.index.EdgeKeyIndexTableWrapper;
import edu.jhuapl.tinkerpop.tables.index.IndexMetadataTableWrapper;
import edu.jhuapl.tinkerpop.tables.index.VertexKeyIndexTableWrapper;
/**
* Internal class gathering together instances of
* objects needed for various AccumuloGraph components.
*/
public class GlobalInstances {
private final AccumuloGraphConfiguration config;
private final MultiTableBatchWriter mtbw;
private final ElementCaches caches;
public GlobalInstances(AccumuloGraphConfiguration config,
MultiTableBatchWriter mtbw, ElementCaches caches) {
this.config = config;
this.mtbw = mtbw;
this.caches = caches;
}
public AccumuloGraphConfiguration getConfig() {
return config;
}
public MultiTableBatchWriter getMtbw() {
return mtbw;
}
public VertexTableWrapper getVertexWrapper() {
return new VertexTableWrapper(this);
}
public EdgeTableWrapper getEdgeWrapper() {
return new EdgeTableWrapper(this);
}
public VertexKeyIndexTableWrapper getVertexKeyIndexWrapper() {
return new VertexKeyIndexTableWrapper(this);
}
public EdgeKeyIndexTableWrapper getEdgeKeyIndexWrapper() {
return new EdgeKeyIndexTableWrapper(this);
}
public IndexMetadataTableWrapper getIndexMetadataWrapper() {
return new IndexMetadataTableWrapper(this);
}
public <T extends Element> ElementTableWrapper getElementWrapper(Class<T> clazz) {
if (Vertex.class.equals(clazz)) {
return getVertexWrapper();
} else if (Edge.class.equals(clazz)) {
return getEdgeWrapper();
} else {
throw new AccumuloGraphException("Unrecognized class: "+clazz);
}
}
public <T extends Element> BaseKeyIndexTableWrapper getKeyIndexTableWrapper(Class<T> clazz) {
if (Vertex.class.equals(clazz)) {
return getVertexKeyIndexWrapper();
} else if (Edge.class.equals(clazz)) {
return getEdgeKeyIndexWrapper();
} else {
throw new AccumuloGraphException("Unrecognized class: "+clazz);
}
}
public ElementCaches getCaches() {
return caches;
}
/**
* Flush the writer, if autoflush is enabled.
*/
public void checkedFlush() {
if (config.getAutoFlush()) {
try {
mtbw.flush();
} catch (MutationsRejectedException e) {
throw new AccumuloGraphException(e);
}
}
}
}

View File

@@ -14,7 +14,6 @@
*/
package edu.jhuapl.tinkerpop;
import java.io.Closeable;
import java.util.Iterator;
import java.util.Map.Entry;
@@ -23,15 +22,17 @@ import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Value;
import org.apache.accumulo.core.util.PeekingIterator;
import com.tinkerpop.blueprints.CloseableIterable;
import com.tinkerpop.blueprints.Element;
public abstract class ScannerIterable<T extends Element> implements Iterable<T>, Closeable {
/**
* TODO
*/
public abstract class ScannerIterable<T extends Element> implements CloseableIterable<T> {
AccumuloGraph parent;
ScannerBase scanner;
private ScannerBase scanner;
ScannerIterable(AccumuloGraph parent, ScannerBase scanner) {
this.parent = parent;
public ScannerIterable(ScannerBase scanner) {
this.scanner = scanner;
}
@@ -55,10 +56,10 @@ public abstract class ScannerIterable<T extends Element> implements Iterable<T>,
close();
}
class ScannerIterator implements Iterator<T> {
PeekingIterator<Entry<Key,Value>> iterator;
private class ScannerIterator implements Iterator<T> {
private PeekingIterator<Entry<Key,Value>> iterator;
ScannerIterator(PeekingIterator<Entry<Key,Value>> iterator) {
private ScannerIterator(PeekingIterator<Entry<Key,Value>> iterator) {
this.iterator = iterator;
}
@@ -76,7 +77,5 @@ public abstract class ScannerIterable<T extends Element> implements Iterable<T>,
public void remove() {
throw new UnsupportedOperationException();
}
}
}

View File

@@ -12,7 +12,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop;
package edu.jhuapl.tinkerpop.cache;
import java.util.concurrent.TimeUnit;

View File

@@ -0,0 +1,77 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.cache;
import com.tinkerpop.blueprints.Edge;
import com.tinkerpop.blueprints.Element;
import com.tinkerpop.blueprints.Vertex;
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration;
import edu.jhuapl.tinkerpop.AccumuloGraphException;
/**
* Utility class wrapping element caches.
*/
public class ElementCaches {
private ElementCache<Vertex> vertexCache;
private ElementCache<Edge> edgeCache;
public ElementCaches(AccumuloGraphConfiguration config) {
if (config.getVertexCacheEnabled()) {
vertexCache = new ElementCache<Vertex>(config.getVertexCacheSize(),
config.getVertexCacheTimeout());
}
if (config.getEdgeCacheEnabled()) {
edgeCache = new ElementCache<Edge>(config.getEdgeCacheSize(),
config.getEdgeCacheTimeout());
}
}
public <T extends Element> void cache(T element, Class<T> clazz) {
if (pick(clazz) != null) {
pick(clazz).cache(element);
}
}
public <T extends Element> T retrieve(Object id, Class<T> clazz) {
return pick(clazz) != null ? pick(clazz).retrieve(id) : null;
}
public <T extends Element> void remove(Object id, Class<T> clazz) {
if (pick(clazz) != null) {
pick(clazz).remove(id);
}
}
public <T extends Element> void clear(Class<T> clazz) {
if (pick(clazz) != null) {
pick(clazz).clear();
}
}
@SuppressWarnings("unchecked")
private <T extends Element> ElementCache<T> pick(Class<T> clazz) {
if (Vertex.class.equals(clazz)) {
return (ElementCache<T>) vertexCache;
}
else if (Edge.class.equals(clazz)) {
return (ElementCache<T>) edgeCache;
}
else {
throw new AccumuloGraphException("Unknown element class: "+clazz);
}
}
}

View File

@@ -12,10 +12,14 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop;
package edu.jhuapl.tinkerpop.cache;
import java.util.HashMap;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Set;
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration;
/**
* Cache for storing element properties.
@@ -34,6 +38,14 @@ public class PropertyCache {
this.values = new HashMap<String, TimedValue>();
}
public boolean containsKey(String key) {
return values.containsKey(key);
}
public Set<String> keySet() {
return values.keySet();
}
public void put(String key, Object value) {
Integer timeout = getTimeout(key);
@@ -42,9 +54,17 @@ public class PropertyCache {
return;
}
values.put(key, new TimedValue(value,
timeout != null ? System.currentTimeMillis() + timeout : null));
System.currentTimeMillis() + timeout));
}
public void putAll(Map<String, Object> entries) {
if(entries==null)return;
for (Entry<String,Object> ent : entries.entrySet()) {
put(ent.getKey(), ent.getValue());
}
}
@SuppressWarnings("unchecked")
public <T> T get(String key) {
long now = System.currentTimeMillis();
@@ -71,6 +91,11 @@ public class PropertyCache {
values.clear();
}
@Override
public String toString() {
return values.toString();
}
/**
* Return the timeout for the given key.
* Checks for a key-specific timeout
@@ -106,5 +131,10 @@ public class PropertyCache {
public Long getExpiry() {
return expiry;
}
@Override
public String toString() {
return "[" + value + ", " + expiry + "]";
}
}
}

View File

@@ -24,6 +24,8 @@ import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
import edu.jhuapl.tinkerpop.AccumuloGraph;
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration;
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration.InstanceType;
import edu.jhuapl.tinkerpop.AccumuloGraphException;
import edu.jhuapl.tinkerpop.Constants;
public class EdgeInputFormat extends InputFormatBase<Text,Edge> {
@@ -64,8 +66,7 @@ public class EdgeInputFormat extends InputFormatBase<Text,Edge> {
}
parent = AccumuloGraph.open(conf.getConfiguration());
} catch (AccumuloException e) {
// TODO Auto-generated catch block
e.printStackTrace();
throw new AccumuloGraphException(e);
}
}
@@ -84,10 +85,10 @@ public class EdgeInputFormat extends InputFormatBase<Text,Edge> {
String eid = currentKey.getRow().toString();
String colf = currentKey.getColumnFamily().toString();
switch (colf) {
case AccumuloGraph.SLABEL:
case Constants.LABEL:
currentK.set(eid);
edge.prepareId(eid);
String[] ids = currentKey.getColumnQualifier().toString().split(parent.IDDELIM);
String[] ids = currentKey.getColumnQualifier().toString().split(Constants.ID_DELIM);
edge.setSourceId(ids[1]);
edge.setDestId(ids[0]);
edge.setLabel(AccumuloByteSerializer.deserialize(entry.getValue().get()).toString());
@@ -109,7 +110,7 @@ public class EdgeInputFormat extends InputFormatBase<Text,Edge> {
public static void setAccumuloGraphConfiguration(Job job, AccumuloGraphConfiguration cfg) throws AccumuloSecurityException {
EdgeInputFormat.setConnectorInfo(job, cfg.getUser(), new PasswordToken(cfg.getPassword()));
EdgeInputFormat.setInputTableName(job, cfg.getEdgeTable());
EdgeInputFormat.setInputTableName(job, cfg.getEdgeTableName());
if (cfg.getInstanceType().equals(InstanceType.Mock)) {
EdgeInputFormat.setMockInstance(job, cfg.getInstanceName());
} else {

View File

@@ -25,6 +25,7 @@ import com.tinkerpop.blueprints.Element;
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration;
import edu.jhuapl.tinkerpop.AccumuloGraphException;
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration.InstanceType;
public class ElementOutputFormat extends OutputFormat<NullWritable,Element> {
@@ -69,7 +70,7 @@ public class ElementOutputFormat extends OutputFormat<NullWritable,Element> {
return new NullOutputFormat<Text,Mutation>().getOutputCommitter(context);
}
class ElementRecordWriter extends RecordWriter<NullWritable,Element> {
static class ElementRecordWriter extends RecordWriter<NullWritable,Element> {
AccumuloGraphConfiguration config;
protected ElementRecordWriter(TaskAttemptContext context) {
@@ -92,9 +93,9 @@ public class ElementOutputFormat extends OutputFormat<NullWritable,Element> {
try {
if (bw == null) {
if (ele instanceof MapReduceVertex) {
bw = config.getConnector().createBatchWriter(config.getVertexTable(), config.getBatchWriterConfig());
bw = config.getConnector().createBatchWriter(config.getVertexTableName(), config.getBatchWriterConfig());
} else {
bw = config.getConnector().createBatchWriter(config.getEdgeTable(), config.getBatchWriterConfig());
bw = config.getConnector().createBatchWriter(config.getEdgeTableName(), config.getBatchWriterConfig());
}
}
@@ -105,9 +106,7 @@ public class ElementOutputFormat extends OutputFormat<NullWritable,Element> {
bw.addMutation(mut);
} catch (TableNotFoundException | AccumuloException | AccumuloSecurityException e) {
// TODO Auto-generated catch block
e.printStackTrace();
throw new RuntimeException(e);
throw new AccumuloGraphException(e);
}
}
@@ -117,7 +116,7 @@ public class ElementOutputFormat extends OutputFormat<NullWritable,Element> {
try {
bw.close();
} catch (MutationsRejectedException e) {
e.printStackTrace();
throw new AccumuloGraphException(e);
}
}
}

View File

@@ -64,6 +64,7 @@ public abstract class MapReduceElement implements Element, WritableComparable<Ma
return id;
}
@SuppressWarnings("unchecked")
@Override
public <T> T getProperty(String key) {

View File

@@ -23,7 +23,9 @@ import com.tinkerpop.blueprints.Vertex;
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
import edu.jhuapl.tinkerpop.AccumuloGraph;
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration;
import edu.jhuapl.tinkerpop.AccumuloGraphException;
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration.InstanceType;
import edu.jhuapl.tinkerpop.Constants;
public class VertexInputFormat extends InputFormatBase<Text,Vertex> {
static AccumuloGraphConfiguration conf;
@@ -65,10 +67,8 @@ public class VertexInputFormat extends InputFormatBase<Text,Vertex> {
parent = AccumuloGraph.open(conf.getConfiguration());
} catch (AccumuloException e) {
// TODO Auto-generated catch block
e.printStackTrace();
throw new AccumuloGraphException(e);
}
}
@Override
@@ -85,17 +85,17 @@ public class VertexInputFormat extends InputFormatBase<Text,Vertex> {
String vid = currentKey.getRow().toString();
String colf = currentKey.getColumnFamily().toString();
switch (colf) {
case AccumuloGraph.SLABEL:
case Constants.LABEL:
currentK.set(vid);
vertex.prepareId(vid);
break;
case AccumuloGraph.SINEDGE:
String[] parts = currentKey.getColumnQualifier().toString().split(AccumuloGraph.IDDELIM);
case Constants.IN_EDGE:
String[] parts = currentKey.getColumnQualifier().toString().split(Constants.ID_DELIM);
String label = new String(entry.getValue().get());
vertex.prepareEdge(parts[1], parts[0], label, vid);
break;
case AccumuloGraph.SOUTEDGE:
parts = currentKey.getColumnQualifier().toString().split(AccumuloGraph.IDDELIM);
case Constants.OUT_EDGE:
parts = currentKey.getColumnQualifier().toString().split(Constants.ID_DELIM);
label = new String(entry.getValue().get());
vertex.prepareEdge(parts[1], vid, label, parts[0]);
break;
@@ -116,7 +116,7 @@ public class VertexInputFormat extends InputFormatBase<Text,Vertex> {
public static void setAccumuloGraphConfiguration(Job job, AccumuloGraphConfiguration cfg) throws AccumuloSecurityException {
VertexInputFormat.setConnectorInfo(job, cfg.getUser(), new PasswordToken(cfg.getPassword()));
VertexInputFormat.setInputTableName(job, cfg.getVertexTable());
VertexInputFormat.setInputTableName(job, cfg.getVertexTableName());
if (cfg.getInstanceType().equals(InstanceType.Mock)) {
VertexInputFormat.setMockInstance(job, cfg.getInstanceName());
} else {

View File

@@ -0,0 +1,22 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.mutator;
import org.apache.accumulo.core.data.Mutation;
public interface Mutator {
public Iterable<Mutation> create();
}

View File

@@ -0,0 +1,53 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.mutator;
import java.util.LinkedList;
import java.util.List;
import org.apache.accumulo.core.client.BatchDeleter;
import org.apache.accumulo.core.client.BatchWriter;
import org.apache.accumulo.core.client.MutationsRejectedException;
import org.apache.accumulo.core.data.Range;
import com.tinkerpop.blueprints.Element;
import edu.jhuapl.tinkerpop.AccumuloGraphException;
public class Mutators {
public static void apply(BatchWriter writer, Mutator mut) {
try {
writer.addMutations(mut.create());
} catch (MutationsRejectedException e) {
throw new AccumuloGraphException(e);
}
}
public static void deleteElementRanges(BatchDeleter deleter, Element... elements) {
List<Range> ranges = new LinkedList<Range>();
for (Element element : elements) {
ranges.add(new Range(element.getId().toString()));
}
deleter.setRanges(ranges);
try {
deleter.delete();
} catch (Exception e) {
throw new AccumuloGraphException(e);
}
}
}

View File

@@ -0,0 +1,41 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.mutator.edge;
import com.tinkerpop.blueprints.Direction;
import com.tinkerpop.blueprints.Edge;
import edu.jhuapl.tinkerpop.mutator.Mutator;
public abstract class BaseEdgeMutator implements Mutator {
protected final String id;
protected final String outVertexId;
protected final String inVertexId;
protected final String label;
public BaseEdgeMutator(Edge edge) {
this(edge.getId().toString(),
edge.getVertex(Direction.OUT).getId().toString(),
edge.getVertex(Direction.IN).getId().toString(),
edge.getLabel());
}
public BaseEdgeMutator(String id, String outVertexId, String inVertexId, String label) {
this.id = id;
this.outVertexId = outVertexId;
this.inVertexId = inVertexId;
this.label = label;
}
}

View File

@@ -0,0 +1,78 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.mutator.edge;
import org.apache.accumulo.core.data.Mutation;
import com.google.common.collect.Lists;
import com.tinkerpop.blueprints.Edge;
import edu.jhuapl.tinkerpop.Constants;
public class EdgeEndpointsMutator {
private EdgeEndpointsMutator() {
}
public static class Add extends BaseEdgeMutator {
public Add(Edge edge) {
super(edge);
}
public Add(String id, String outVertexId, String inVertexId, String label) {
super(id, outVertexId, inVertexId, label);
}
@Override
public Iterable<Mutation> create() {
Mutation in = new Mutation(inVertexId);
in.put(Constants.IN_EDGE.getBytes(),
(outVertexId + Constants.ID_DELIM + id).getBytes(),
(Constants.ID_DELIM + label).getBytes());
Mutation out = new Mutation(outVertexId);
out.put(Constants.OUT_EDGE.getBytes(),
(inVertexId + Constants.ID_DELIM + id).getBytes(),
(Constants.ID_DELIM + label).getBytes());
return Lists.newArrayList(in, out);
}
}
public static class Delete extends BaseEdgeMutator {
public Delete(Edge edge) {
super(edge);
}
public Delete(String id, String outVertexId, String inVertexId, String label) {
super(id, outVertexId, inVertexId, label);
}
@Override
public Iterable<Mutation> create() {
Mutation in = new Mutation(inVertexId);
in.putDelete(Constants.IN_EDGE.getBytes(),
(outVertexId + Constants.ID_DELIM + id).getBytes());
Mutation out = new Mutation(outVertexId);
out.putDelete(Constants.OUT_EDGE.getBytes(),
(inVertexId + Constants.ID_DELIM + id).getBytes());
return Lists.newArrayList(in, out);
}
}
}

View File

@@ -0,0 +1,67 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.mutator.edge;
import org.apache.accumulo.core.data.Mutation;
import com.google.common.collect.Lists;
import com.tinkerpop.blueprints.Edge;
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
import edu.jhuapl.tinkerpop.Constants;
public final class EdgeMutator {
public static class Add extends BaseEdgeMutator {
public Add(Edge edge) {
super(edge);
}
public Add(String id, String outVertexId, String inVertexId, String label) {
super(id, outVertexId, inVertexId, label);
}
@Override
public Iterable<Mutation> create() {
Mutation m = new Mutation(id);
m.put(Constants.LABEL.getBytes(),
(inVertexId + Constants.ID_DELIM + outVertexId).getBytes(),
AccumuloByteSerializer.serialize(label));
return Lists.newArrayList(m);
}
}
public static class Delete extends BaseEdgeMutator {
public Delete(Edge edge) {
super(edge);
}
public Delete(String id, String outVertexId, String inVertexId, String label) {
super(id, outVertexId, inVertexId, label);
}
@Override
public Iterable<Mutation> create() {
Mutation m = new Mutation(id);
m.putDelete(Constants.LABEL.getBytes(),
(inVertexId + Constants.ID_DELIM + outVertexId).getBytes());
return Lists.newArrayList(m);
}
}
}

View File

@@ -0,0 +1,76 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.mutator.index;
import org.apache.accumulo.core.data.Mutation;
import com.google.common.collect.Lists;
import com.tinkerpop.blueprints.Element;
import edu.jhuapl.tinkerpop.Constants;
import edu.jhuapl.tinkerpop.Constants.IndexMetadataEntryType;
import edu.jhuapl.tinkerpop.mutator.Mutator;
/**
* Mutators for index metadata table entries.
*/
public class IndexMetadataMutator {
private IndexMetadataMutator() { }
public static class Add implements Mutator {
private final String key;
private final Class<? extends Element> elementClass;
private final IndexMetadataEntryType entryType;
public Add(String key, Class<? extends Element> elementClass,
IndexMetadataEntryType entryType) {
this.key = key;
this.elementClass = elementClass;
this.entryType = entryType;
}
@Override
public Iterable<Mutation> create() {
Mutation m = new Mutation(key);
m.put(entryType.name().getBytes(),
elementClass.getName().getBytes(), Constants.EMPTY);
return Lists.newArrayList(m);
}
}
public static class Delete implements Mutator {
private final String key;
private final Class<? extends Element> elementClass;
private final IndexMetadataEntryType entryType;
public Delete(String indexName, Class<? extends Element> elementClass,
IndexMetadataEntryType entryType) {
this.key = indexName;
this.elementClass = elementClass;
this.entryType = entryType;
}
@Override
public Iterable<Mutation> create() {
Mutation m = new Mutation(key);
m.putDelete(entryType.name().getBytes(),
elementClass.getName().getBytes());
return Lists.newArrayList(m);
}
}
}

View File

@@ -0,0 +1,75 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.mutator.index;
import org.apache.accumulo.core.data.Mutation;
import com.google.common.collect.Lists;
import com.tinkerpop.blueprints.Element;
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
import edu.jhuapl.tinkerpop.Constants;
import edu.jhuapl.tinkerpop.mutator.Mutator;
/**
* Mutators for vertex/edge index tables.
*/
public class IndexValueMutator {
private IndexValueMutator() { }
public static class Add implements Mutator {
private final Element element;
private final String key;
private final Object value;
public Add(Element element, String key, Object value) {
this.element = element;
this.key = key;
this.value = value;
}
@Override
public Iterable<Mutation> create() {
byte[] bytes = AccumuloByteSerializer.serialize(value);
Mutation m = new Mutation(bytes);
m.put(key.getBytes(), element.getId().toString()
.getBytes(), Constants.EMPTY);
return Lists.newArrayList(m);
}
}
public static class Delete implements Mutator {
private final Element element;
private final String key;
private final Object value;
public Delete(Element element, String key, Object value) {
this.element = element;
this.key = key;
this.value = value;
}
@Override
public Iterable<Mutation> create() {
byte[] bytes = AccumuloByteSerializer.serialize(value);
Mutation m = new Mutation(bytes);
m.putDelete(key, element.getId().toString());
return Lists.newArrayList(m);
}
}
}

View File

@@ -0,0 +1,28 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.mutator.property;
import edu.jhuapl.tinkerpop.mutator.Mutator;
public abstract class BasePropertyMutator implements Mutator {
protected final String id;
protected final String key;
public BasePropertyMutator(String id, String key) {
this.id = id;
this.key = key;
}
}

View File

@@ -0,0 +1,34 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.mutator.property;
import org.apache.accumulo.core.data.Mutation;
import com.google.common.collect.Lists;
import edu.jhuapl.tinkerpop.Constants;
public class ClearPropertyMutator extends BasePropertyMutator {
public ClearPropertyMutator(String id, String key) {
super(id, key);
}
@Override
public Iterable<Mutation> create() {
Mutation m = new Mutation(id);
m.putDelete(key.getBytes(), Constants.EMPTY);
return Lists.newArrayList(m);
}
}

View File

@@ -0,0 +1,39 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.mutator.property;
import org.apache.accumulo.core.data.Mutation;
import com.google.common.collect.Lists;
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
import edu.jhuapl.tinkerpop.Constants;
public class WritePropertyMutator extends BasePropertyMutator {
private final Object value;
public WritePropertyMutator(String id, String key, Object value) {
super(id, key);
this.value = value;
}
@Override
public Iterable<Mutation> create() {
byte[] bytes = AccumuloByteSerializer.serialize(value);
Mutation m = new Mutation(id);
m.put(key.getBytes(), Constants.EMPTY, bytes);
return Lists.newArrayList(m);
}
}

View File

@@ -0,0 +1,38 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.mutator.vertex;
import org.apache.accumulo.core.data.Mutation;
import com.google.common.collect.Lists;
import edu.jhuapl.tinkerpop.Constants;
import edu.jhuapl.tinkerpop.mutator.Mutator;
public final class AddVertexMutator implements Mutator {
private final String id;
public AddVertexMutator(String id) {
this.id = id;
}
@Override
public Iterable<Mutation> create() {
Mutation m = new Mutation(id);
m.put(Constants.LABEL.getBytes(),
Constants.EXISTS.getBytes(), Constants.EMPTY);
return Lists.newArrayList(m);
}
}

View File

@@ -0,0 +1,33 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.parser;
import edu.jhuapl.tinkerpop.AccumuloEdge;
import edu.jhuapl.tinkerpop.GlobalInstances;
/**
* Edge-specific index parser.
*/
public class EdgeIndexParser extends ElementIndexParser<AccumuloEdge> {
public EdgeIndexParser(GlobalInstances globals) {
super(globals);
}
@Override
protected AccumuloEdge instantiate(String id) {
return new AccumuloEdge(globals, id);
}
}

View File

@@ -0,0 +1,70 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.parser;
import java.util.Map.Entry;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Value;
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
import edu.jhuapl.tinkerpop.AccumuloEdge;
import edu.jhuapl.tinkerpop.AccumuloGraphException;
import edu.jhuapl.tinkerpop.AccumuloVertex;
import edu.jhuapl.tinkerpop.Constants;
import edu.jhuapl.tinkerpop.GlobalInstances;
/**
* TODO
*/
public class EdgeParser extends ElementParser<AccumuloEdge> {
public EdgeParser(GlobalInstances globals) {
super(globals);
}
@Override
public AccumuloEdge parse(String id, Iterable<Entry<Key,Value>> entries) {
AccumuloEdge edge = makeEdge(id, entries);
setInMemoryProperties(edge, entries);
return edge;
}
/**
* Make and return an edge object. If the entries
* contain label/endpoint information, set those too.
* @param id
* @param entries
* @return
*/
private AccumuloEdge makeEdge(String id, Iterable<Entry<Key,Value>> entries) {
for (Entry<Key, Value> entry : entries) {
String cf = entry.getKey().getColumnFamily().toString();
if (Constants.LABEL.equals(cf)) {
String cq = entry.getKey().getColumnQualifier().toString();
String[] parts = cq.split(Constants.ID_DELIM);
String inVertexId = parts[0];
String outVertexId = parts[1];
String label = AccumuloByteSerializer.deserialize(entry.getValue().get());
return new AccumuloEdge(globals, id,
new AccumuloVertex(globals, inVertexId),
new AccumuloVertex(globals, outVertexId), label);
}
}
// This should not happen.
throw new AccumuloGraphException("Unable to parse edge from entries");
}
}

View File

@@ -0,0 +1,72 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.parser;
import java.util.Iterator;
import java.util.Map.Entry;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Value;
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
import edu.jhuapl.tinkerpop.AccumuloElement;
import edu.jhuapl.tinkerpop.AccumuloGraphException;
import edu.jhuapl.tinkerpop.GlobalInstances;
/**
* Parser for elements based on an index table.
*/
public abstract class ElementIndexParser<T extends AccumuloElement>
implements EntryParser<T> {
protected final GlobalInstances globals;
public ElementIndexParser(GlobalInstances globals) {
this.globals = globals;
}
@Override
public T parse(Iterable<Entry<Key, Value>> entries) {
Iterator<Entry<Key, Value>> it = entries.iterator();
if (it.hasNext()) {
Entry<Key, Value> entry = it.next();
if (it.hasNext()) {
throw new AccumuloGraphException("Unexpected multiple entries for index table");
}
String id = entry.getKey().getColumnQualifierData().toString();
T element = instantiate(id);
// While we're here, read the property key/value.
String key = entry.getKey().getColumnFamily().toString();
Object value = AccumuloByteSerializer.deserialize(entry.getKey().getRow().getBytes());
element.setPropertyInMemory(key, value);
return element;
}
else {
throw new AccumuloGraphException("No index table entries found");
}
}
/**
* Instantiate an object to be returned.
* @param id
* @return
*/
protected abstract T instantiate(String id);
}

View File

@@ -0,0 +1,65 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.parser;
import java.util.Map;
import java.util.Map.Entry;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Value;
import edu.jhuapl.tinkerpop.AccumuloElement;
import edu.jhuapl.tinkerpop.GlobalInstances;
/**
* TODO
*/
public abstract class ElementParser<T extends AccumuloElement> implements EntryParser<T> {
protected GlobalInstances globals;
public ElementParser(GlobalInstances globals) {
this.globals = globals;
}
@Override
public T parse(Iterable<Entry<Key, Value>> entries) {
String id = entries.iterator().next().getKey().getRow().toString();
return parse(id, entries);
}
/**
* Given the element id and set of entries,
* create the type of element. This can leverage
* the property loader, etc.
* @param id
* @param entries
* @return
*/
public abstract T parse(String id, Iterable<Entry<Key, Value>> entries);
/**
* Parse out the property entries and set them for the given element.
* @param element
* @param entries
*/
protected void setInMemoryProperties(T element, Iterable<Entry<Key, Value>> entries) {
Map<String, Object> props = new PropertyParser().parse(entries);
if(props == null) return;
for (Entry<String, Object> ent : props.entrySet()) {
element.setPropertyInMemory(ent.getKey(), ent.getValue());
}
}
}

View File

@@ -0,0 +1,28 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.parser;
import java.util.Map.Entry;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Value;
/**
* TODO
*/
public interface EntryParser<T> {
public T parse(Iterable<Entry<Key, Value>> entries);
}

View File

@@ -0,0 +1,44 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.parser;
import com.tinkerpop.blueprints.Element;
import com.tinkerpop.blueprints.IndexableGraph;
import com.tinkerpop.blueprints.KeyIndexableGraph;
/**
* An indexed item. For {@link IndexableGraph},
* the key is the index name. For {@link KeyIndexableGraph}
* the key is the indexed key.
* @author Michael Lieberman
*
*/
public class IndexedItem {
private final String key;
private final Class<? extends Element> elementClass;
public IndexedItem(String key, Class<? extends Element> elementClass) {
this.key = key;
this.elementClass = elementClass;
}
public String getKey() {
return key;
}
public Class<? extends Element> getElementClass() {
return elementClass;
}
}

View File

@@ -0,0 +1,91 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.parser;
import java.util.ArrayList;
import java.util.List;
import java.util.Map.Entry;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Value;
import com.tinkerpop.blueprints.Edge;
import com.tinkerpop.blueprints.Element;
import com.tinkerpop.blueprints.IndexableGraph;
import com.tinkerpop.blueprints.KeyIndexableGraph;
import com.tinkerpop.blueprints.Vertex;
import edu.jhuapl.tinkerpop.AccumuloGraphException;
/**
* Entry parser for index metadata. The format
* is the same for both {@link IndexableGraph}
* and {@link KeyIndexableGraph} functionality.
* For the former, this parser returns names of
* indexes. For the latter, the parser returns
* indexed keys.
*/
public class IndexedItemsListParser implements EntryParser<List<IndexedItem>> {
private final Class<? extends Element> elementClass;
/**
* Constructor to return all items regardless
* of element class.
*/
public IndexedItemsListParser() {
this(Element.class);
}
/**
* Create a parser for items of the specified element class.
* This may be Vertex, Edge, or Element.
* @param elementClass
*/
public IndexedItemsListParser(Class<? extends Element> elementClass) {
// Validate element class.
if (!Vertex.class.equals(elementClass) &&
!Edge.class.equals(elementClass) &&
!Element.class.equals(elementClass)) {
throw new IllegalArgumentException("elementClass must be Vertex, Edge or Element");
}
this.elementClass = elementClass;
}
@SuppressWarnings("unchecked")
@Override
public List<IndexedItem> parse(Iterable<Entry<Key,Value>> entries) {
List<IndexedItem> items = new ArrayList<IndexedItem>();
for (Entry<Key, Value> entry : entries) {
Class<? extends Element> clazz;
try {
clazz = (Class<? extends Element>) Class.forName(entry.getKey()
.getColumnQualifier().toString());
} catch (ClassNotFoundException e) {
throw new AccumuloGraphException(e);
}
if (Element.class.equals(elementClass) ||
elementClass.equals(clazz)) {
IndexedItem item = new IndexedItem(entry.getKey()
.getRow().toString(), clazz);
items.add(item);
}
}
return items;
}
}

View File

@@ -0,0 +1,66 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.parser;
import java.util.HashMap;
import java.util.Map;
import java.util.Map.Entry;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Value;
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
import edu.jhuapl.tinkerpop.Constants;
/**
* TODO
*/
public class PropertyParser implements EntryParser<Map<String, Object>> {
@Override
public Map<String,Object> parse(Iterable<Entry<Key,Value>> entries) {
Map<String, Object> props = null;
for (Entry<Key, Value> entry : entries) {
if (props == null) {
props = new HashMap<String, Object>();
}
Key key = entry.getKey();
if (!isMetaKey(key)) {
String attr = key.getColumnFamily().toString();
Object value = AccumuloByteSerializer.deserialize(entry.getValue().get());
props.put(attr, value);
}
}
return props;
}
/**
* Test whether the given Accumulo key represents a
* metadata key (e.g. existence, edge endpoint, etc),
* rather than a property.
* @param key
* @return
*/
private static boolean isMetaKey(Key key) {
String cf = key.getColumnFamily().toString();
return Constants.LABEL.equals(cf) ||
Constants.IN_EDGE.equals(cf) ||
Constants.OUT_EDGE.equals(cf);
}
}

View File

@@ -0,0 +1,33 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.parser;
import edu.jhuapl.tinkerpop.AccumuloVertex;
import edu.jhuapl.tinkerpop.GlobalInstances;
/**
* Vertex-specific index parser.
*/
public class VertexIndexParser extends ElementIndexParser<AccumuloVertex> {
public VertexIndexParser(GlobalInstances globals) {
super(globals);
}
@Override
protected AccumuloVertex instantiate(String id) {
return new AccumuloVertex(globals, id);
}
}

View File

@@ -12,29 +12,29 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop;
package edu.jhuapl.tinkerpop.parser;
import java.util.Map.Entry;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Value;
public enum EntryLocation {
import edu.jhuapl.tinkerpop.AccumuloVertex;
import edu.jhuapl.tinkerpop.GlobalInstances;
Row, ColF, ColQ, Value;
/**
* TODO
*/
public class VertexParser extends ElementParser<AccumuloVertex> {
public String extract(Entry<Key,Value> entry) {
switch (this) {
case Row:
return entry.getKey().getRow().toString();
case ColF:
return entry.getKey().getColumnFamily().toString();
case ColQ:
return entry.getKey().getColumnQualifier().toString();
case Value:
return new String(entry.getValue().get());
default:
throw new AccumuloGraphException("Unexpected type: " + this);
}
public VertexParser(GlobalInstances globals) {
super(globals);
}
@Override
public AccumuloVertex parse(String id, Iterable<Entry<Key,Value>> entries) {
AccumuloVertex vertex = new AccumuloVertex(globals, id);
setInMemoryProperties(vertex, entries);
return vertex;
}
}

View File

@@ -0,0 +1,90 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.tables;
import java.util.Collections;
import java.util.Map.Entry;
import org.apache.accumulo.core.client.BatchDeleter;
import org.apache.accumulo.core.client.BatchScanner;
import org.apache.accumulo.core.client.BatchWriter;
import org.apache.accumulo.core.client.Scanner;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Range;
import org.apache.accumulo.core.data.Value;
import edu.jhuapl.tinkerpop.AccumuloGraphException;
import edu.jhuapl.tinkerpop.GlobalInstances;
/**
* Table wrapper with common functionality.
*/
public abstract class BaseTableWrapper {
protected GlobalInstances globals;
private String tableName;
public BaseTableWrapper(GlobalInstances globals, String tableName) {
this.globals = globals;
this.tableName = tableName;
}
protected Scanner getScanner() {
try {
return globals.getConfig().getConnector().createScanner(tableName,
globals.getConfig().getAuthorizations());
} catch (Exception e) {
throw new AccumuloGraphException(e);
}
}
protected BatchScanner getBatchScanner() {
try {
BatchScanner scanner = globals.getConfig().getConnector().createBatchScanner(tableName,
globals.getConfig().getAuthorizations(), globals.getConfig().getQueryThreads());
scanner.setRanges(Collections.singletonList(new Range()));
return scanner;
} catch (Exception e) {
throw new AccumuloGraphException(e);
}
}
protected BatchWriter getWriter() {
try {
return globals.getMtbw().getBatchWriter(tableName);
} catch (Exception e) {
throw new AccumuloGraphException(e);
}
}
protected BatchDeleter getDeleter() {
try {
return globals.getConfig().getConnector().createBatchDeleter(tableName,
globals.getConfig().getAuthorizations(), globals.getConfig().getMaxWriteThreads(),
globals.getConfig().getBatchWriterConfig());
} catch (Exception e) {
throw new AccumuloGraphException(e);
}
}
public void dump() {
System.out.println("Dump of table "+tableName+":");
Scanner s = getScanner();
for (Entry<Key, Value> entry : s) {
System.out.println(" "+entry);
}
}
}

View File

@@ -0,0 +1,177 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.tables.core;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import java.util.Map.Entry;
import java.util.regex.Pattern;
import org.apache.accumulo.core.client.BatchScanner;
import org.apache.accumulo.core.client.IteratorSetting;
import org.apache.accumulo.core.client.Scanner;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Range;
import org.apache.accumulo.core.data.Value;
import org.apache.accumulo.core.iterators.user.RegExFilter;
import org.apache.accumulo.core.util.PeekingIterator;
import org.apache.hadoop.io.Text;
import com.tinkerpop.blueprints.CloseableIterable;
import com.tinkerpop.blueprints.Edge;
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
import edu.jhuapl.tinkerpop.AccumuloEdge;
import edu.jhuapl.tinkerpop.AccumuloGraphException;
import edu.jhuapl.tinkerpop.AccumuloGraphUtils;
import edu.jhuapl.tinkerpop.AccumuloVertex;
import edu.jhuapl.tinkerpop.Constants;
import edu.jhuapl.tinkerpop.GlobalInstances;
import edu.jhuapl.tinkerpop.ScannerIterable;
import edu.jhuapl.tinkerpop.mutator.Mutators;
import edu.jhuapl.tinkerpop.mutator.edge.EdgeMutator;
import edu.jhuapl.tinkerpop.parser.EdgeParser;
/**
* Wrapper around {@link Edge} tables.
*/
public class EdgeTableWrapper extends ElementTableWrapper {
public EdgeTableWrapper(GlobalInstances globals) {
super(globals, globals.getConfig().getEdgeTableName());
}
/**
* Write the given edge to the edge table. Does not
* currently write the edge's properties.
*
* <p/>Note: This only adds the edge information. Vertex
* endpoint information needs to be written to the vertex
* table via {@link VertexTableWrapper}.
* @param edge
*/
public void writeEdge(Edge edge) {
Mutators.apply(getWriter(), new EdgeMutator.Add(edge));
globals.checkedFlush();
}
public void deleteEdge(Edge edge) {
Mutators.apply(getWriter(), new EdgeMutator.Delete(edge));
globals.checkedFlush();
}
public CloseableIterable<Edge> getEdges() {
Scanner scan = getScanner();
scan.fetchColumnFamily(new Text(Constants.LABEL));
if (globals.getConfig().getPreloadedProperties() != null) {
for (String key : globals.getConfig().getPreloadedProperties()) {
scan.fetchColumnFamily(new Text(key));
}
}
final EdgeParser parser = new EdgeParser(globals);
return new ScannerIterable<Edge>(scan) {
@Override
public Edge next(PeekingIterator<Entry<Key, Value>> iterator) {
// TODO could also check local cache before creating a new instance?
String rowId = iterator.peek().getKey().getRow().toString();
List<Entry<Key, Value>> entries =
new ArrayList<Entry<Key, Value>>();
// MDL 05 Jan 2014: Why is this equalsIgnoreCase??
while (iterator.peek() != null && rowId.equalsIgnoreCase(iterator
.peek().getKey().getRow().toString())) {
entries.add(iterator.next());
}
AccumuloEdge edge = parser.parse(rowId, entries);
globals.getCaches().cache(edge, Edge.class);
return edge;
}
};
}
public Iterable<Edge> getEdges(String key, Object value) {
AccumuloGraphUtils.nullCheckProperty(key, value);
if (key.equalsIgnoreCase("label")) {
key = Constants.LABEL;
}
BatchScanner scan = getBatchScanner();
scan.fetchColumnFamily(new Text(key));
byte[] val = AccumuloByteSerializer.serialize(value);
if (val[0] != AccumuloByteSerializer.SERIALIZABLE) {
IteratorSetting is = new IteratorSetting(10, "filter", RegExFilter.class);
RegExFilter.setRegexs(is, null, null, null, Pattern.quote(new String(val)), false);
scan.addScanIterator(is);
return new ScannerIterable<Edge>(scan) {
@Override
public Edge next(PeekingIterator<Entry<Key,Value>> iterator) {
Key k = iterator.next().getKey();
if (k.getColumnFamily().toString().equals(Constants.LABEL)) {
String[] vals = k.getColumnQualifier().toString().split(Constants.ID_DELIM);
return new AccumuloEdge(globals, k.getRow().toString(),
new AccumuloVertex(globals, vals[0]),
new AccumuloVertex(globals, vals[1]), null);
}
return new AccumuloEdge(globals, k.getRow().toString());
}
};
} else {
// TODO
throw new UnsupportedOperationException("Filtering on binary data not currently supported.");
}
}
public void loadEndpointsAndLabel(AccumuloEdge edge) {
Scanner s = getScanner();
try {
s.setRange(new Range(edge.getId().toString()));
s.fetchColumnFamily(new Text(Constants.LABEL));
Iterator<Entry<Key,Value>> iter = s.iterator();
if (!iter.hasNext()) {
dump();
throw new AccumuloGraphException("Unable to find edge row: "+edge);
}
Entry<Key, Value> entry = iter.next();
String cq = entry.getKey().getColumnQualifier().toString();
String[] ids = cq.split(Constants.ID_DELIM);
String label = AccumuloByteSerializer.deserialize(entry.getValue().get());
edge.setVertices(new AccumuloVertex(globals, ids[0]),
new AccumuloVertex(globals, ids[1]));
edge.setLabel(label);
} finally {
s.close();
}
}
}

View File

@@ -0,0 +1,226 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.tables.core;
import java.util.HashSet;
import java.util.Iterator;
import java.util.Map;
import java.util.Set;
import java.util.Map.Entry;
import org.apache.accumulo.core.client.BatchWriter;
import org.apache.accumulo.core.client.IteratorSetting;
import org.apache.accumulo.core.client.Scanner;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Range;
import org.apache.accumulo.core.data.Value;
import org.apache.accumulo.core.iterators.user.RegExFilter;
import org.apache.hadoop.io.Text;
import com.tinkerpop.blueprints.Element;
import com.tinkerpop.blueprints.util.StringFactory;
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
import edu.jhuapl.tinkerpop.Constants;
import edu.jhuapl.tinkerpop.GlobalInstances;
import edu.jhuapl.tinkerpop.mutator.property.ClearPropertyMutator;
import edu.jhuapl.tinkerpop.mutator.property.WritePropertyMutator;
import edu.jhuapl.tinkerpop.mutator.Mutators;
import edu.jhuapl.tinkerpop.parser.PropertyParser;
import edu.jhuapl.tinkerpop.tables.BaseTableWrapper;
/**
* Wrapper around tables with operations
* common to {@link Element}s.
*/
public abstract class ElementTableWrapper extends BaseTableWrapper {
private BatchWriter writer;
public ElementTableWrapper(GlobalInstances globals, String tableName) {
super(globals, tableName);
writer = super.getWriter();
}
/**
* Give a single instance of the writer for this table.
*/
@Override
protected BatchWriter getWriter() {
return writer;
}
/**
* Read the given property from the backing table
* for the given element id.
* @param id
* @param key
* @return
*/
public <V> V readProperty(Element element, String key) {
Scanner s = getScanner();
s.setRange(new Range(element.getId().toString()));
Text colf = StringFactory.LABEL.equals(key)
? new Text(Constants.LABEL) : new Text(key);
s.fetchColumnFamily(colf);
V value = null;
Iterator<Entry<Key, Value>> iter = s.iterator();
if (iter.hasNext()) {
value = AccumuloByteSerializer.deserialize(iter.next().getValue().get());
}
s.close();
return value;
}
/**
* Read all properties for the given element
* from the backing table.
* If the element has no properties, return an empty Map.
* If the element does not exist, return null.
* @param element
* @return
*/
public Map<String, Object> readAllProperties(Element element) {
return readProperties(element, null);
}
/**
* Read the given properties for the given element.
* If propertyKeys is null, read all properties.
* If the element has no properties, return an empty Map.
* If the element does not exist, return null.
* @param id
* @param propertyKeys
* @return
*/
public Map<String, Object> readProperties(Element element, String[] propertyKeys) {
Scanner s = getScanner();
s.setRange(Range.exact((String) element.getId()));
// If propertyKeys is null, we read everything.
// Otherwise, limit to the given attributes.
if (propertyKeys != null) {
s.fetchColumnFamily(new Text(Constants.LABEL));
for (String key : propertyKeys) {
s.fetchColumnFamily(new Text(key));
}
}
Map<String, Object> props = new PropertyParser().parse(s);
s.close();
return props;
}
/**
* Return true if the element with given id exists.
* @param id
* @return
*/
public boolean elementExists(String id) {
Scanner scan = null;
try {
scan = getScanner();
scan.setRange(Range.exact(id));
scan.fetchColumnFamily(new Text(Constants.LABEL));
return new PropertyParser().parse(scan) != null;
} finally {
if (scan != null) {
scan.close();
}
}
}
/**
* Get all property keys for the given element id.
* @param id
* @return
*/
public Set<String> readPropertyKeys(Element element) {
Scanner s = getScanner();
s.setRange(new Range(element.getId().toString()));
Set<String> keys = new HashSet<String>();
for (Entry<Key, Value> entry : s) {
String cf = entry.getKey().getColumnFamily().toString();
keys.add(cf);
}
s.close();
// Remove some special keys.
keys.remove(Constants.IN_EDGE);
keys.remove(Constants.LABEL);
keys.remove(Constants.OUT_EDGE);
return keys;
}
/**
* Delete the property entry from property table.
* @param id
* @param key
*/
public void clearProperty(Element element, String key) {
Mutators.apply(getWriter(),
new ClearPropertyMutator(element.getId().toString(), key));
globals.checkedFlush();
}
/**
* Write the given property to the property table.
* @param id
* @param key
* @param value
*/
public void writeProperty(Element element, String key, Object value) {
Mutators.apply(getWriter(),
new WritePropertyMutator(element.getId().toString(),
key, value));
globals.checkedFlush();
}
/**
* Add custom iterator to the given scanner so that
* it will only return keys with value corresponding to an edge.
* @param scan
* @param labels
*/
protected void applyEdgeLabelValueFilter(Scanner scan, String... labels) {
StringBuilder regex = new StringBuilder();
for (String lab : labels) {
if (regex.length() != 0)
regex.append("|");
regex.append(".*"+Constants.ID_DELIM+"\\Q").append(lab).append("\\E$");
}
IteratorSetting is = new IteratorSetting(10, "edgeValueFilter", RegExFilter.class);
RegExFilter.setRegexs(is, null, null, null, regex.toString(), false);
scan.addScanIterator(is);
}
public void close() {
// TODO?
}
}

View File

@@ -0,0 +1,269 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.tables.core;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Map.Entry;
import java.util.regex.Pattern;
import org.apache.accumulo.core.client.BatchDeleter;
import org.apache.accumulo.core.client.BatchScanner;
import org.apache.accumulo.core.client.IteratorSetting;
import org.apache.accumulo.core.client.Scanner;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Range;
import org.apache.accumulo.core.data.Value;
import org.apache.accumulo.core.iterators.user.RegExFilter;
import org.apache.accumulo.core.util.PeekingIterator;
import org.apache.hadoop.io.Text;
import com.tinkerpop.blueprints.CloseableIterable;
import com.tinkerpop.blueprints.Direction;
import com.tinkerpop.blueprints.Edge;
import com.tinkerpop.blueprints.Vertex;
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
import edu.jhuapl.tinkerpop.AccumuloEdge;
import edu.jhuapl.tinkerpop.AccumuloElement;
import edu.jhuapl.tinkerpop.AccumuloGraphException;
import edu.jhuapl.tinkerpop.AccumuloGraphUtils;
import edu.jhuapl.tinkerpop.AccumuloVertex;
import edu.jhuapl.tinkerpop.Constants;
import edu.jhuapl.tinkerpop.GlobalInstances;
import edu.jhuapl.tinkerpop.ScannerIterable;
import edu.jhuapl.tinkerpop.mutator.vertex.AddVertexMutator;
import edu.jhuapl.tinkerpop.mutator.Mutator;
import edu.jhuapl.tinkerpop.mutator.Mutators;
import edu.jhuapl.tinkerpop.mutator.edge.EdgeEndpointsMutator;
import edu.jhuapl.tinkerpop.parser.VertexParser;
/**
* Wrapper around {@link Vertex} tables.
*/
public class VertexTableWrapper extends ElementTableWrapper {
public VertexTableWrapper(GlobalInstances globals) {
super(globals, globals.getConfig().getVertexTableName());
}
/**
* Write a vertex with the given id.
* Note: This does not currently write the vertex's properties.
* @param vertex
*/
public void writeVertex(Vertex vertex) {
Mutators.apply(getWriter(), new AddVertexMutator(vertex.getId().toString()));
globals.checkedFlush();
}
/**
* Remove the given vertex.
* Note: This uses a BatchDeleter rather than {@link Mutator}
* because it is more efficient.
* @param vertex
*/
public void deleteVertex(Vertex vertex) {
BatchDeleter deleter = null;
try {
deleter = getDeleter();
deleter.setRanges(Arrays.asList(Range.exact((String) vertex.getId())));
deleter.delete();
} catch (Exception e) {
throw new AccumuloGraphException(e);
} finally {
if (deleter != null) {
deleter.close();
}
}
}
/**
* Write edge endpoint information to the vertex table.
* @param edge
*/
public void writeEdgeEndpoints(Edge edge) {
Mutators.apply(getWriter(), new EdgeEndpointsMutator.Add(edge));
globals.checkedFlush();
}
public void deleteEdgeEndpoints(Edge edge) {
Mutators.apply(getWriter(), new EdgeEndpointsMutator.Delete(edge));
globals.checkedFlush();
}
public CloseableIterable<Edge> getEdges(Vertex vertex, Direction direction,
String... labels) {
Scanner scan = getScanner();
scan.setRange(new Range(vertex.getId().toString()));
if (direction.equals(Direction.IN)) {
scan.fetchColumnFamily(new Text(Constants.IN_EDGE));
} else if (direction.equals(Direction.OUT)) {
scan.fetchColumnFamily(new Text(Constants.OUT_EDGE));
} else {
scan.fetchColumnFamily(new Text(Constants.IN_EDGE));
scan.fetchColumnFamily(new Text(Constants.OUT_EDGE));
}
if (labels.length > 0) {
applyEdgeLabelValueFilter(scan, labels);
}
return new ScannerIterable<Edge>(scan) {
@Override
public Edge next(PeekingIterator<Entry<Key,Value>> iterator) {
// TODO better use of information readily available...
// TODO could also check local cache before creating a new
// instance?
Entry<Key,Value> kv = iterator.next();
String[] parts = kv.getKey().getColumnQualifier().toString().split(Constants.ID_DELIM);
String label = (new String(kv.getValue().get())).split(Constants.ID_DELIM)[1];
AccumuloEdge edge;
if (kv.getKey().getColumnFamily().toString().equalsIgnoreCase(Constants.IN_EDGE)) {
edge = new AccumuloEdge(globals, parts[1],
new AccumuloVertex(globals, kv.getKey().getRow().toString()),
new AccumuloVertex(globals, parts[0]), label);
} else {
edge = new AccumuloEdge(globals, parts[1],
new AccumuloVertex(globals, parts[0]),
new AccumuloVertex(globals, kv.getKey().getRow().toString()), label);
}
globals.getCaches().cache(edge, Edge.class);
return edge;
}
};
}
public Iterable<Vertex> getVertices(Vertex vertex, Direction direction, String... labels) {
Scanner scan = getScanner();
scan.setRange(new Range(vertex.getId().toString()));
if (direction.equals(Direction.IN)) {
scan.fetchColumnFamily(new Text(Constants.IN_EDGE));
} else if (direction.equals(Direction.OUT)) {
scan.fetchColumnFamily(new Text(Constants.OUT_EDGE));
} else {
scan.fetchColumnFamily(new Text(Constants.IN_EDGE));
scan.fetchColumnFamily(new Text(Constants.OUT_EDGE));
}
if (labels != null && labels.length > 0) {
applyEdgeLabelValueFilter(scan, labels);
}
return new ScannerIterable<Vertex>(scan) {
@Override
public Vertex next(PeekingIterator<Entry<Key,Value>> iterator) {
// TODO better use of information readily available...
// TODO could also check local cache before creating a new
// instance?
String[] parts = iterator.next().getKey().getColumnQualifier()
.toString().split(Constants.ID_DELIM);
AccumuloVertex vertex = new AccumuloVertex(globals, parts[0]);
globals.getCaches().cache(vertex, Vertex.class);
return vertex;
}
};
}
public CloseableIterable<Vertex> getVertices() {
return getVerticesInRange(null, null);
}
public CloseableIterable<Vertex> getVerticesInRange(Object fromId, Object toId) {
Scanner scan = getScanner();
scan.setRange(new Range(fromId != null ? fromId.toString() : null,
toId != null ? toId.toString() : null));
scan.fetchColumnFamily(new Text(Constants.LABEL));
if (globals.getConfig().getPreloadedProperties() != null) {
for (String key : globals.getConfig().getPreloadedProperties()) {
scan.fetchColumnFamily(new Text(key));
}
}
final VertexParser parser = new VertexParser(globals);
return new ScannerIterable<Vertex>(scan) {
@Override
public Vertex next(PeekingIterator<Entry<Key, Value>> iterator) {
// TODO could also check local cache before creating a new instance?
String rowId = iterator.peek().getKey().getRow().toString();
List<Entry<Key, Value>> entries =
new ArrayList<Entry<Key, Value>>();
while (iterator.peek() != null && rowId.equals(iterator
.peek().getKey().getRow().toString())) {
entries.add(iterator.next());
}
AccumuloVertex vertex = parser.parse(rowId, entries);
globals.getCaches().cache(vertex, Vertex.class);
return vertex;
}
};
}
public Iterable<Vertex> getVertices(String key, Object value) {
AccumuloGraphUtils.validateProperty(key, value);
byte[] val = AccumuloByteSerializer.serialize(value);
if (val[0] != AccumuloByteSerializer.SERIALIZABLE) {
BatchScanner scan = getBatchScanner();
scan.fetchColumnFamily(new Text(key));
IteratorSetting is = new IteratorSetting(10, "filter", RegExFilter.class);
RegExFilter.setRegexs(is, null, null, null, Pattern.quote(new String(val)), false);
scan.addScanIterator(is);
return new ScannerIterable<Vertex>(scan) {
@Override
public Vertex next(PeekingIterator<Entry<Key,Value>> iterator) {
Entry<Key, Value> kv = iterator.next();
String key = kv.getKey().getColumnFamily().toString();
Object value = AccumuloByteSerializer.deserialize(kv.getValue().get());
Vertex v = globals.getCaches().retrieve(kv.getKey().getRow().toString(), Vertex.class);
if (v == null) {
v = new AccumuloVertex(globals, kv.getKey().getRow().toString());
}
((AccumuloElement) v).setPropertyInMemory(key, value);
globals.getCaches().cache(v, Vertex.class);
return v;
}
};
} else {
// TODO
throw new UnsupportedOperationException("Filtering on binary data not currently supported.");
}
}
}

View File

@@ -0,0 +1,179 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.tables.index;
import java.util.Arrays;
import java.util.Collections;
import java.util.Map.Entry;
import java.util.regex.Pattern;
import org.apache.accumulo.core.client.BatchDeleter;
import org.apache.accumulo.core.client.BatchWriter;
import org.apache.accumulo.core.client.IteratorSetting;
import org.apache.accumulo.core.client.Scanner;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Range;
import org.apache.accumulo.core.data.Value;
import org.apache.accumulo.core.iterators.user.RegExFilter;
import org.apache.accumulo.core.util.PeekingIterator;
import org.apache.hadoop.io.Text;
import com.tinkerpop.blueprints.CloseableIterable;
import com.tinkerpop.blueprints.Element;
import com.tinkerpop.blueprints.IndexableGraph;
import com.tinkerpop.blueprints.Vertex;
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
import edu.jhuapl.tinkerpop.AccumuloElement;
import edu.jhuapl.tinkerpop.AccumuloGraphException;
import edu.jhuapl.tinkerpop.AccumuloGraphUtils;
import edu.jhuapl.tinkerpop.GlobalInstances;
import edu.jhuapl.tinkerpop.ScannerIterable;
import edu.jhuapl.tinkerpop.mutator.Mutators;
import edu.jhuapl.tinkerpop.mutator.index.IndexValueMutator;
import edu.jhuapl.tinkerpop.parser.EdgeIndexParser;
import edu.jhuapl.tinkerpop.parser.ElementIndexParser;
import edu.jhuapl.tinkerpop.parser.VertexIndexParser;
import edu.jhuapl.tinkerpop.tables.BaseTableWrapper;
/**
* Wrapper around index tables containing properties
* and values.
*/
public abstract class BaseIndexValuesTableWrapper extends BaseTableWrapper {
protected final Class<? extends Element> elementType;
protected BaseIndexValuesTableWrapper(GlobalInstances globals,
Class<? extends Element> elementType, String tableName) {
super(globals, tableName);
this.elementType = elementType;
}
/**
* Return class of this index.
* @return
*/
public Class<? extends Element> getElementType() {
return elementType;
}
/**
* Add the property to this index, if autoindexing is enabled
* and/or the given key has indexing enabled.
* @param element
* @param key
* @param value
*/
public void setPropertyForIndex(Element element, String key, Object value) {
setPropertyForIndex(element, key, value, false);
}
/**
* Add the property to this index.
*
* <p/>Note that this requires a round-trip to Accumulo to see
* if the property exists if the provided key has an index.
* So for best performance, create indices after bulk ingest.
* <p/>If the force parameter is true, set the property regardless
* of whether indexing is enabled for the given key. This is needed
* for {@link IndexableGraph} operations.
* @param element
* @param key
* @param value
* @param force
*/
public void setPropertyForIndex(Element element, String key, Object value,
boolean force) {
AccumuloGraphUtils.validateProperty(key, value);
if (force || globals.getConfig().getAutoIndex() ||
globals.getIndexMetadataWrapper()
.getIndexedKeys(elementType).contains(key)) {
BatchWriter writer = getWriter();
Object oldValue = element.getProperty(key);
if (oldValue != null && !oldValue.equals(value)) {
Mutators.apply(writer, new IndexValueMutator.Delete(element, key, oldValue));
}
Mutators.apply(writer, new IndexValueMutator.Add(element, key, value));
globals.checkedFlush();
}
}
/**
* Remove property from the index.
* @param element
* @param key
* @param value
*/
public void removePropertyFromIndex(Element element, String key, Object value) {
if (value != null) {
Mutators.apply(getWriter(), new IndexValueMutator.Delete(element, key, value));
globals.checkedFlush();
}
}
/**
* Get elements with the key/value pair.
* @param key
* @param value
* @return
*/
@SuppressWarnings("unchecked")
public <T extends Element> CloseableIterable<T> readElementsFromIndex(String key, Object value) {
Scanner scan = getScanner();
byte[] id = AccumuloByteSerializer.serialize(value);
scan.setRange(Range.exact(new Text(id)));
scan.fetchColumnFamily(new Text(key));
final ElementIndexParser<? extends AccumuloElement> parser =
Vertex.class.equals(elementType) ? new VertexIndexParser(globals) :
new EdgeIndexParser(globals);
return new ScannerIterable<T>(scan) {
@Override
public T next(PeekingIterator<Entry<Key,Value>> iterator) {
return (T) parser.parse(Arrays.asList(iterator.next()));
}
};
}
/**
* Remove the given element's properties from the index.
* @param element
*/
public void removeElementFromIndex(Element element) {
BatchDeleter deleter = null;
try {
deleter = getDeleter();
deleter.setRanges(Collections.singleton(new Range()));
IteratorSetting is = new IteratorSetting(10, "getEdgeFilter", RegExFilter.class);
RegExFilter.setRegexs(is, null, null,
"^"+Pattern.quote(element.getId().toString())+"$", null, false);
deleter.addScanIterator(is);
deleter.delete();
deleter.close();
} catch (Exception e) {
throw new AccumuloGraphException(e);
} finally {
if (deleter != null) {
deleter.close();
}
}
}
}

View File

@@ -0,0 +1,77 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.tables.index;
import com.tinkerpop.blueprints.CloseableIterable;
import com.tinkerpop.blueprints.Edge;
import com.tinkerpop.blueprints.Element;
import com.tinkerpop.blueprints.Vertex;
import edu.jhuapl.tinkerpop.AccumuloGraphException;
import edu.jhuapl.tinkerpop.GlobalInstances;
import edu.jhuapl.tinkerpop.tables.core.EdgeTableWrapper;
import edu.jhuapl.tinkerpop.tables.core.ElementTableWrapper;
import edu.jhuapl.tinkerpop.tables.core.VertexTableWrapper;
/**
* Base class for key index tables.
*/
public abstract class BaseKeyIndexTableWrapper extends BaseIndexValuesTableWrapper {
protected BaseKeyIndexTableWrapper(GlobalInstances globals,
Class<? extends Element> elementType, String tableName) {
super(globals, elementType, tableName);
}
/**
* Rebuild this index for the given table.
* @param table
* @param key
*/
public void rebuildIndex(String key, Class<? extends Element> elementClass) {
ElementTableWrapper wrapper = globals.getElementWrapper(elementClass);
if (wrapper instanceof VertexTableWrapper) {
CloseableIterable<Vertex> iter = ((VertexTableWrapper) wrapper).getVertices();
for (Vertex v : iter) {
rebuild(wrapper, v, key);
}
iter.close();
}
else if (wrapper instanceof EdgeTableWrapper) {
CloseableIterable<Edge> iter = ((EdgeTableWrapper) wrapper).getEdges();
for (Edge e : iter) {
rebuild(wrapper, e, key);
}
iter.close();
}
else {
throw new AccumuloGraphException("Unexpected table wrapper: "+wrapper.getClass());
}
globals.checkedFlush();
}
/**
* Add given element to index for the given key.
* @param element
* @param key
*/
private void rebuild(ElementTableWrapper wrapper,
Element element, String key) {
Object value = wrapper.readProperty(element, key);
if (value != null) {
setPropertyForIndex(element, key, value);
}
}
}

View File

@@ -0,0 +1,86 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.tables.index;
import java.util.Arrays;
import java.util.Map.Entry;
import org.apache.accumulo.core.client.Scanner;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Range;
import org.apache.accumulo.core.data.Value;
import org.apache.accumulo.core.util.PeekingIterator;
import org.apache.hadoop.io.Text;
import com.tinkerpop.blueprints.Edge;
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
import edu.jhuapl.tinkerpop.AccumuloEdge;
import edu.jhuapl.tinkerpop.GlobalInstances;
import edu.jhuapl.tinkerpop.ScannerIterable;
import edu.jhuapl.tinkerpop.parser.EdgeIndexParser;
/**
* Wrapper around {@link Edge} index table.
*/
public class EdgeKeyIndexTableWrapper extends BaseKeyIndexTableWrapper {
public EdgeKeyIndexTableWrapper(GlobalInstances globals) {
super(globals, Edge.class, globals.getConfig()
.getEdgeKeyIndexTableName());
}
/**
* Retrieve edges from the index table based
* on the given key/value.
* @param key
* @param value
* @return
*/
public Iterable<Edge> getEdges(String key, Object value) {
Scanner s = getScanner();
Text row = new Text(AccumuloByteSerializer.serialize(value));
s.setRange(Range.exact(row));
s.fetchColumnFamily(new Text(key));
final EdgeIndexParser parser = new EdgeIndexParser(globals);
return new ScannerIterable<Edge>(s) {
@Override
public Edge next(PeekingIterator<Entry<Key, Value>> iterator) {
Entry<Key, Value> entry = iterator.next();
AccumuloEdge e = parser.parse(Arrays.asList(entry));
// Check if we have it cached already, in which
// case use the cached version.
AccumuloEdge cached = (AccumuloEdge) globals.getCaches()
.retrieve(e.getId(), Edge.class);
if (cached != null) {
for (String key : e.getPropertyKeysInMemory()) {
cached.setPropertyInMemory(key, e.getPropertyInMemory(key));
}
return cached;
}
// We don't have it, so cache the new one and return it.
globals.getCaches().cache(e, Edge.class);
return e;
}
};
}
}

View File

@@ -0,0 +1,170 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.tables.index;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import org.apache.accumulo.core.client.Scanner;
import org.apache.hadoop.io.Text;
import com.tinkerpop.blueprints.Element;
import com.tinkerpop.blueprints.Index;
import com.tinkerpop.blueprints.IndexableGraph;
import com.tinkerpop.blueprints.KeyIndexableGraph;
import com.tinkerpop.blueprints.util.ExceptionFactory;
import edu.jhuapl.tinkerpop.AccumuloIndex;
import edu.jhuapl.tinkerpop.Constants.IndexMetadataEntryType;
import edu.jhuapl.tinkerpop.GlobalInstances;
import edu.jhuapl.tinkerpop.mutator.Mutators;
import edu.jhuapl.tinkerpop.mutator.index.IndexMetadataMutator;
import edu.jhuapl.tinkerpop.parser.IndexedItem;
import edu.jhuapl.tinkerpop.parser.IndexedItemsListParser;
import edu.jhuapl.tinkerpop.tables.BaseTableWrapper;
/**
* Stores metadata, in particular the indexed keys
* for {@link KeyIndexableGraph}, and the list of
* named indexes for {@link IndexableGraph}.
*/
public class IndexMetadataTableWrapper extends BaseTableWrapper {
public IndexMetadataTableWrapper(GlobalInstances globals) {
super(globals, globals.getConfig().getIndexMetadataTableName());
}
//////// Methods for KeyIndexableGraph ////////
public void writeKeyMetadataEntry(String key, Class<? extends Element> clazz) {
Mutators.apply(getWriter(), new IndexMetadataMutator.Add(key, clazz,
IndexMetadataEntryType.__INDEX_KEY__));
}
public void clearKeyMetadataEntry(String key, Class<? extends Element> clazz) {
Mutators.apply(getWriter(), new IndexMetadataMutator.Delete(key, clazz,
IndexMetadataEntryType.__INDEX_KEY__));
}
public <T extends Element> Set<String> getIndexedKeys(Class<T> elementClass) {
if (elementClass == null) {
throw ExceptionFactory.classForElementCannotBeNull();
}
IndexedItemsListParser parser = new IndexedItemsListParser(elementClass);
Scanner scan = null;
try {
scan = getScanner();
scan.fetchColumnFamily(new Text(IndexMetadataEntryType.__INDEX_KEY__.name()));
Set<String> keys = new HashSet<String>();
for (IndexedItem item : parser.parse(scan)) {
keys.add(item.getKey());
}
return keys;
} finally {
if (scan != null) {
scan.close();
}
}
}
//////// Methods for IndexableGraph ////////
@SuppressWarnings({"rawtypes", "unchecked"})
public Iterable<Index<? extends Element>> getIndices() {
List<Index<? extends Element>> indexes = new ArrayList<Index<? extends Element>>();
IndexedItemsListParser parser = new IndexedItemsListParser();
Scanner scan = null;
try {
scan = getScanner();
scan.fetchColumnFamily(new Text(IndexMetadataEntryType.__INDEX_NAME__.name()));
for (IndexedItem item : parser.parse(scan)) {
indexes.add(new AccumuloIndex(globals,
item.getKey(), item.getElementClass()));
}
return indexes;
} finally {
if (scan != null) {
scan.close();
}
}
}
public <T extends Element> Index<T> getIndex(String indexName,
Class<T> indexClass) {
IndexedItemsListParser parser = new IndexedItemsListParser();
Scanner scan = null;
try {
scan = getScanner();
scan.fetchColumnFamily(new Text(IndexMetadataEntryType.__INDEX_NAME__.name()));
for (IndexedItem item : parser.parse(scan)) {
if (item.getKey().equals(indexName)) {
if (item.getElementClass().equals(indexClass)) {
return new AccumuloIndex<T>(globals, indexName,
indexClass);
}
else {
throw ExceptionFactory.indexDoesNotSupportClass(indexName, indexClass);
}
}
}
return null;
} finally {
if (scan != null) {
scan.close();
}
}
}
public <T extends Element> Index<T> createIndex(String indexName,
Class<T> indexClass) {
for (Index<?> index : getIndices()) {
if (index.getIndexName().equals(indexName)) {
throw ExceptionFactory.indexAlreadyExists(indexName);
}
}
writeIndexNameEntry(indexName, indexClass);
return new AccumuloIndex<T>(globals, indexName, indexClass);
}
private void writeIndexNameEntry(String indexName,
Class<? extends Element> indexClass) {
Mutators.apply(getWriter(), new IndexMetadataMutator.Add(indexName,
indexClass, IndexMetadataEntryType.__INDEX_NAME__));
}
public void clearIndexNameEntry(String indexName,
Class<? extends Element> indexClass) {
Mutators.apply(getWriter(), new IndexMetadataMutator.Delete(indexName,
indexClass, IndexMetadataEntryType.__INDEX_NAME__));
}
}

View File

@@ -0,0 +1,32 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.tables.index;
import com.tinkerpop.blueprints.Element;
import com.tinkerpop.blueprints.IndexableGraph;
import edu.jhuapl.tinkerpop.GlobalInstances;
/**
* Wrapper around a named index table (for {@link IndexableGraph}).
*/
public class NamedIndexTableWrapper extends BaseIndexValuesTableWrapper {
public NamedIndexTableWrapper(GlobalInstances globals,
Class<? extends Element> elementType, String indexName) {
super(globals, elementType,
globals.getConfig().getNamedIndexTableName(indexName));
}
}

View File

@@ -0,0 +1,85 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.tables.index;
import java.util.Arrays;
import java.util.Map.Entry;
import org.apache.accumulo.core.client.Scanner;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Range;
import org.apache.accumulo.core.data.Value;
import org.apache.accumulo.core.util.PeekingIterator;
import org.apache.hadoop.io.Text;
import com.tinkerpop.blueprints.Vertex;
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
import edu.jhuapl.tinkerpop.AccumuloVertex;
import edu.jhuapl.tinkerpop.GlobalInstances;
import edu.jhuapl.tinkerpop.ScannerIterable;
import edu.jhuapl.tinkerpop.parser.VertexIndexParser;
/**
* Wrapper around {@link Vertex} index table.
*/
public class VertexKeyIndexTableWrapper extends BaseKeyIndexTableWrapper {
public VertexKeyIndexTableWrapper(GlobalInstances globals) {
super(globals, Vertex.class, globals.getConfig()
.getVertexKeyIndexTableName());
}
/**
* Use the index to retrieve vertices with the
* given key/value.
* @param key
* @param value
*/
public Iterable<Vertex> getVertices(String key, Object value) {
Scanner s = getScanner();
Text row = new Text(AccumuloByteSerializer.serialize(value));
s.setRange(Range.exact(row));
s.fetchColumnFamily(new Text(key));
final VertexIndexParser parser = new VertexIndexParser(globals);
return new ScannerIterable<Vertex>(s) {
@Override
public Vertex next(PeekingIterator<Entry<Key, Value>> iterator) {
Entry<Key, Value> entry = iterator.next();
AccumuloVertex v = parser.parse(Arrays.asList(entry));
// Check if we have it cached already, in which
// case use the cached version.
AccumuloVertex cached = (AccumuloVertex) globals.getCaches()
.retrieve(v.getId(), Vertex.class);
if (cached != null) {
for (String key : v.getPropertyKeysInMemory()) {
cached.setPropertyInMemory(key, v.getPropertyInMemory(key));
}
return cached;
}
// We don't have it, so cache the new one and return it.
globals.getCaches().cache(v, Vertex.class);
return v;
}
};
}
}

View File

@@ -0,0 +1,6 @@
log4j.rootLogger=INFO, stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c - %m%n

View File

@@ -43,8 +43,7 @@ public class AccumuloBulkIngesterTest {
ingester.addEdge("A", "B", "edge").add("P3", "V3").finish();
ingester.shutdown(true);
cfg.setClear(false);
AccumuloGraph graph = new AccumuloGraph(cfg);
AccumuloGraph graph = new AccumuloGraph(cfg.clone().setClear(false));
Vertex v1 = graph.getVertex("A");
assertNotNull(v1);

View File

@@ -68,6 +68,34 @@ public class AccumuloElementTest {
graph.shutdown();
}
@Test
public void testGetVerticesInRange() {
AccumuloGraph graph = (AccumuloGraph) AccumuloGraphTestUtils
.makeGraph("testGetVerticesInRange");
assertEquals(0, count(graph.getVerticesInRange(null, null)));
for (int i = 0; i < 10; i++) {
graph.addVertex(id(i));
}
assertEquals(10, count(graph.getVerticesInRange(null, null)));
assertEquals(0, count(graph.getVerticesInRange(id(20), null)));
assertEquals(10, count(graph.getVerticesInRange(null, id(20))));
assertEquals(0, count(graph.getVerticesInRange(id(20), id(20))));
for (int i = 0; i < 10; i++) {
assertEquals(i+1, count(graph.getVerticesInRange(null, id(i))));
assertEquals(10-i, count(graph.getVerticesInRange(id(i), null)));
assertEquals(1, count(graph.getVerticesInRange(id(i), id(i))));
}
graph.shutdown();
}
private static String id(int idNum) {
return String.format("%08d", idNum);
}
private static int count(Iterable<?> iter) {
int count = 0;
for (@SuppressWarnings("unused") Object obj : iter) {

View File

@@ -29,6 +29,8 @@ import org.junit.Test;
import com.tinkerpop.blueprints.GraphFactory;
import com.tinkerpop.blueprints.Vertex;
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration.InstanceType;
public class AccumuloGraphConfigurationTest {
@Test
@@ -149,14 +151,14 @@ public class AccumuloGraphConfigurationTest {
}
graph.shutdown();
graph = new AccumuloGraph(cfg.setCreate(false));
graph = new AccumuloGraph(cfg.clone().setCreate(false));
assertTrue(graph.isEmpty());
graph.addVertex("A");
graph.addVertex("B");
assertFalse(graph.isEmpty());
graph.shutdown();
graph = new AccumuloGraph(cfg.setClear(true));
graph = new AccumuloGraph(cfg.clone().setClear(true));
assertTrue(graph.isEmpty());
graph.shutdown();
}
@@ -257,4 +259,38 @@ public class AccumuloGraphConfigurationTest {
} catch (Exception e) { }
}
}
@Test
public void testPreloadedProperties() {
// Don't allow "all" and "some" preloaded properties.
AccumuloGraphConfiguration conf = new AccumuloGraphConfiguration();
conf.setPreloadAllProperties(true);
conf.setPreloadedProperties(new String[]{"one", "two", "three"});
try {
conf.validate();
fail();
} catch (Exception e) { }
}
@Test
public void testImmutableConnector() throws Exception {
AccumuloGraphConfiguration cfg = new AccumuloGraphConfiguration().setInstanceType(
InstanceType.Mock).setGraphName("immutableConnector")
.setCreate(true).setAutoFlush(false);
cfg.getConnector();
try {
cfg.setCreate(false);
fail();
} catch (Exception e) { }
try {
cfg.setAutoFlush(true);
fail();
} catch (Exception e) { }
assertTrue(cfg.getCreate());
assertFalse(cfg.getAutoFlush());
}
}

View File

@@ -88,6 +88,7 @@ public class AccumuloGraphTest extends GraphTest {
printTestPerformance("GraphSONReaderTestSuite", this.stopWatch());
}
@Override
public void doTestSuite(final TestSuite testSuite) throws Exception {
String doTest = System.getProperty("testTinkerGraph");
if (doTest == null || doTest.equals("true")) {
@@ -112,12 +113,21 @@ public class AccumuloGraphTest extends GraphTest {
new AccumuloGraphTest().generateGraph();
}
@Override
public void dropGraph(final String graphDirectoryName) {
if (graphDirectoryName != null) {
((AccumuloGraph) generateGraph(graphDirectoryName)).clear();
AccumuloGraphConfiguration cfg = AccumuloGraphTestUtils.generateGraphConfig(graphDirectoryName);
try {
for (String table : cfg.getConnector().tableOperations().list()) {
cfg.getConnector().tableOperations().delete(table);
}
} catch (Exception e) {
throw new AccumuloGraphException(e);
}
}
}
@Override
public Object convertId(final Object id) {
return id.toString();
}

View File

@@ -1,35 +1,32 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop;
import com.tinkerpop.blueprints.Graph;
import com.tinkerpop.blueprints.GraphFactory;
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration.InstanceType;
public class AccumuloGraphTestUtils {
public static AccumuloGraphConfiguration generateGraphConfig(String graphDirectoryName) {
AccumuloGraphConfiguration cfg = new AccumuloGraphConfiguration();
cfg.setInstanceName("instanceName").setZooKeeperHosts("ZookeeperHostsString");
cfg.setUser("root").setPassword("");
cfg.setGraphName(graphDirectoryName).setCreate(true).setAutoFlush(true).setInstanceType(InstanceType.Mock);
return cfg;
}
public static Graph makeGraph(String name) {
return GraphFactory.open(generateGraphConfig(name).getConfiguration());
}
}
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop;
import com.tinkerpop.blueprints.Graph;
import com.tinkerpop.blueprints.GraphFactory;
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration.InstanceType;
public class AccumuloGraphTestUtils {
public static AccumuloGraphConfiguration generateGraphConfig(String graphDirectoryName) {
return new AccumuloGraphConfiguration().setInstanceType(InstanceType.Mini)
.setGraphName(graphDirectoryName).setCreate(true);
}
public static Graph makeGraph(String name) {
return GraphFactory.open(generateGraphConfig(name));
}
}

View File

@@ -2,55 +2,117 @@ package edu.jhuapl.tinkerpop;
import static org.junit.Assert.*;
import java.io.IOException;
import java.util.Map.Entry;
import org.apache.accumulo.core.client.AccumuloException;
import org.apache.accumulo.core.client.AccumuloSecurityException;
import org.apache.accumulo.core.client.Scanner;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Value;
import org.junit.Test;
import com.tinkerpop.blueprints.Edge;
import com.tinkerpop.blueprints.Element;
import com.tinkerpop.blueprints.GraphFactory;
import com.tinkerpop.blueprints.Vertex;
public class AutoIndexTest {
@Test
public void testIndexCreation() throws AccumuloException, AccumuloSecurityException, IOException, InterruptedException {
AccumuloGraph ag = (AccumuloGraph) GraphFactory.open(AccumuloGraphTestUtils.generateGraphConfig("AutoIndexTest").setAutoIndex(true).getConfiguration());
String VERT = "1234";
String KEY = "name";
String VALUE = "bananaman";
public void testVertexAutoIndex() throws Exception {
AccumuloGraph graph = (AccumuloGraph) GraphFactory.open(AccumuloGraphTestUtils
.generateGraphConfig("VertexAutoIndexTest").setAutoIndex(true).getConfiguration());
String id = "1234";
String key = "name";
String value = "bananaman";
Vertex v1 = ag.addVertex(VERT);
v1.setProperty(KEY, VALUE);
Vertex v1 = graph.addVertex(id);
v1.setProperty(key, value);
Scanner scan = ag.getVertexIndexScanner();
for (Entry<Key,Value> kv : scan) {
assertEquals(new String(AccumuloByteSerializer.serialize(VALUE)), kv.getKey().getRow().toString());
assertEquals(KEY, kv.getKey().getColumnFamily().toString());
assertEquals(VERT, kv.getKey().getColumnQualifier().toString());
Iterable<Element> elements = graph.getGlobals()
.getVertexKeyIndexWrapper().readElementsFromIndex(key, value);
int count = 0;
for (Element element : elements) {
assertTrue(element instanceof Vertex);
assertEquals(id, element.getId());
assertEquals(value, element.getProperty(key));
count++;
}
assertEquals(1, count);
graph.removeVertex(v1);
elements = graph.getGlobals()
.getVertexKeyIndexWrapper().readElementsFromIndex(key, value);
assertEquals(0, count(elements));
}
@Test
public void testRegularCreation() throws AccumuloException, AccumuloSecurityException, IOException, InterruptedException {
AccumuloGraph ag = (AccumuloGraph) GraphFactory.open(AccumuloGraphTestUtils.generateGraphConfig("NoAutoIndexTest").getConfiguration());
String VERT = "1234";
String KEY = "name";
String VALUE = "bananaman";
public void testVertexNoAutoIndex() throws Exception {
AccumuloGraph graph = (AccumuloGraph) GraphFactory.open(AccumuloGraphTestUtils
.generateGraphConfig("VertexNoAutoIndexTest").getConfiguration());
String id = "1234";
String key = "name";
String value = "bananaman";
Vertex v1 = ag.addVertex(VERT);
v1.setProperty(KEY, VALUE);
Scanner scan = ag.getVertexIndexScanner();
for (Entry<Key,Value> kv : scan) {
assertTrue(false);
}
Vertex v1 = graph.addVertex(id);
v1.setProperty(key, value);
Iterable<Element> elements = graph.getGlobals()
.getVertexKeyIndexWrapper().readElementsFromIndex(key, value);
assertEquals(0, count(elements));
}
@Test
public void testEdgeAutoIndex() throws Exception {
AccumuloGraph graph = (AccumuloGraph) GraphFactory.open(AccumuloGraphTestUtils
.generateGraphConfig("EdgeAutoIndex").setAutoIndex(true).getConfiguration());
String id1 = "A";
String id2 = "B";
String eid = "X";
String key = "name";
String value = "bananaman";
Vertex v1 = graph.addVertex(id1);
Vertex v2 = graph.addVertex(id2);
Edge e = graph.addEdge(eid, v1, v2, "edge");
e.setProperty(key, value);
Iterable<Element> elements = graph.getGlobals()
.getEdgeKeyIndexWrapper().readElementsFromIndex(key, value);
int count = 0;
for (Element element : elements) {
assertTrue(element instanceof Edge);
assertEquals(eid, element.getId());
assertEquals(value, element.getProperty(key));
count++;
}
assertEquals(1, count);
graph.removeVertex(v1);
elements = graph.getGlobals()
.getEdgeKeyIndexWrapper().readElementsFromIndex(key, value);
assertEquals(0, count(elements));
}
@Test
public void testEdgeNoAutoIndex() throws Exception {
AccumuloGraph graph = (AccumuloGraph) GraphFactory.open(AccumuloGraphTestUtils
.generateGraphConfig("EdgeNoAutoIndexTest").getConfiguration());
String id1 = "A";
String id2 = "B";
String eid = "X";
String key = "name";
String value = "bananaman";
Vertex v1 = graph.addVertex(id1);
Vertex v2 = graph.addVertex(id2);
Edge e = graph.addEdge(eid, v1, v2, "edge");
e.setProperty(key, value);
Iterable<Element> elements = graph.getGlobals()
.getEdgeKeyIndexWrapper().readElementsFromIndex(key, value);
assertEquals(0, count(elements));
}
@SuppressWarnings("unused")
private static int count(Iterable<?> it) {
int count = 0;
for (Object obj : it) {
count++;
}
return count;
}
}

View File

@@ -24,6 +24,8 @@ import com.tinkerpop.blueprints.Graph;
import com.tinkerpop.blueprints.GraphFactory;
import com.tinkerpop.blueprints.Vertex;
import edu.jhuapl.tinkerpop.cache.ElementCache;
public class ElementCacheTest {
@Test

View File

@@ -0,0 +1,273 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop;
import static org.junit.Assert.*;
import org.junit.Test;
import com.google.common.collect.Sets;
import com.tinkerpop.blueprints.Element;
import com.tinkerpop.blueprints.Graph;
import com.tinkerpop.blueprints.GraphFactory;
/**
* Tests related to {@link Element}-based property
* loading and caching.
*/
public class ElementPropertyCachingTest {
private static final int TIMEOUT = 300000;
private static final String NON_CACHED = "noncached";
private static final String CACHED = "cached";
@Test
public void testCachingDisabled() {
AccumuloGraphConfiguration cfg =
AccumuloGraphTestUtils.generateGraphConfig("cachingDisabled");
assertTrue(cfg.getPropertyCacheTimeout(null) <= 0);
assertTrue(cfg.getPropertyCacheTimeout(NON_CACHED) <= 0);
assertTrue(cfg.getPropertyCacheTimeout(CACHED) <= 0);
Graph graph = open(cfg);
load(graph);
AccumuloVertex a = (AccumuloVertex) graph.getVertex("A");
AccumuloVertex b = (AccumuloVertex) graph.getVertex("B");
AccumuloVertex c = (AccumuloVertex) graph.getVertex("C");
assertEquals(null, a.getProperty(NON_CACHED));
assertEquals(true, b.getProperty(NON_CACHED));
assertEquals(null, c.getProperty(NON_CACHED));
assertEquals(null, a.getProperty(CACHED));
assertEquals(null, b.getProperty(CACHED));
assertEquals(true, c.getProperty(CACHED));
assertEquals(null, a.getPropertyCache().get(NON_CACHED));
assertEquals(null, b.getPropertyCache().get(NON_CACHED));
assertEquals(null, c.getPropertyCache().get(NON_CACHED));
assertEquals(null, a.getPropertyCache().get(CACHED));
assertEquals(null, b.getPropertyCache().get(CACHED));
assertEquals(null, c.getPropertyCache().get(CACHED));
assertEquals(Sets.newHashSet(), a.getPropertyCache().keySet());
assertEquals(Sets.newHashSet(), b.getPropertyCache().keySet());
assertEquals(Sets.newHashSet(), c.getPropertyCache().keySet());
a.removeProperty(NON_CACHED);
b.removeProperty(NON_CACHED);
c.removeProperty(NON_CACHED);
a.removeProperty(CACHED);
b.removeProperty(CACHED);
c.removeProperty(CACHED);
assertEquals(null, a.getProperty(NON_CACHED));
assertEquals(null, b.getProperty(NON_CACHED));
assertEquals(null, c.getProperty(NON_CACHED));
assertEquals(null, a.getProperty(CACHED));
assertEquals(null, b.getProperty(CACHED));
assertEquals(null, c.getProperty(CACHED));
assertEquals(null, a.getPropertyCache().get(NON_CACHED));
assertEquals(null, b.getPropertyCache().get(NON_CACHED));
assertEquals(null, c.getPropertyCache().get(NON_CACHED));
assertEquals(null, a.getPropertyCache().get(CACHED));
assertEquals(null, b.getPropertyCache().get(CACHED));
assertEquals(null, c.getPropertyCache().get(CACHED));
assertEquals(Sets.newHashSet(), a.getPropertyCache().keySet());
assertEquals(Sets.newHashSet(), b.getPropertyCache().keySet());
assertEquals(Sets.newHashSet(), c.getPropertyCache().keySet());
graph.shutdown();
}
@Test
public void testSpecificCaching() {
AccumuloGraphConfiguration cfg =
AccumuloGraphTestUtils.generateGraphConfig("getProperty");
cfg.setPropertyCacheTimeout(CACHED, TIMEOUT);
assertTrue(cfg.getPropertyCacheTimeout(null) <= 0);
assertTrue(cfg.getPropertyCacheTimeout(NON_CACHED) <= 0);
assertEquals(TIMEOUT, cfg.getPropertyCacheTimeout(CACHED));
Graph graph = open(cfg);
load(graph);
AccumuloVertex a = (AccumuloVertex) graph.getVertex("A");
AccumuloVertex b = (AccumuloVertex) graph.getVertex("B");
AccumuloVertex c = (AccumuloVertex) graph.getVertex("C");
assertEquals(null, a.getProperty(NON_CACHED));
assertEquals(true, b.getProperty(NON_CACHED));
assertEquals(null, c.getProperty(NON_CACHED));
assertEquals(null, a.getProperty(CACHED));
assertEquals(null, b.getProperty(CACHED));
assertEquals(true, c.getProperty(CACHED));
assertEquals(null, a.getPropertyCache().get(NON_CACHED));
assertEquals(null, b.getPropertyCache().get(NON_CACHED));
assertEquals(null, c.getPropertyCache().get(NON_CACHED));
assertEquals(null, a.getPropertyCache().get(CACHED));
assertEquals(null, b.getPropertyCache().get(CACHED));
assertEquals(true, c.getPropertyCache().get(CACHED));
assertEquals(Sets.newHashSet(), a.getPropertyCache().keySet());
assertEquals(Sets.newHashSet(), b.getPropertyCache().keySet());
assertEquals(Sets.newHashSet(CACHED), c.getPropertyCache().keySet());
a.removeProperty(NON_CACHED);
b.removeProperty(NON_CACHED);
c.removeProperty(NON_CACHED);
a.removeProperty(CACHED);
b.removeProperty(CACHED);
c.removeProperty(CACHED);
assertEquals(null, a.getProperty(NON_CACHED));
assertEquals(null, b.getProperty(NON_CACHED));
assertEquals(null, c.getProperty(NON_CACHED));
assertEquals(null, a.getProperty(CACHED));
assertEquals(null, b.getProperty(CACHED));
assertEquals(null, c.getProperty(CACHED));
assertEquals(null, a.getPropertyCache().get(NON_CACHED));
assertEquals(null, b.getPropertyCache().get(NON_CACHED));
assertEquals(null, c.getPropertyCache().get(NON_CACHED));
assertEquals(null, a.getPropertyCache().get(CACHED));
assertEquals(null, b.getPropertyCache().get(CACHED));
assertEquals(null, c.getPropertyCache().get(CACHED));
assertEquals(Sets.newHashSet(), a.getPropertyCache().keySet());
assertEquals(Sets.newHashSet(), b.getPropertyCache().keySet());
assertEquals(Sets.newHashSet(), c.getPropertyCache().keySet());
graph.shutdown();
}
@Test
public void testAllCaching() {
AccumuloGraphConfiguration cfg =
AccumuloGraphTestUtils.generateGraphConfig("setProperty");
cfg.setPropertyCacheTimeout(null, TIMEOUT);
cfg.setPropertyCacheTimeout(CACHED, TIMEOUT);
assertEquals(TIMEOUT, cfg.getPropertyCacheTimeout(null));
assertEquals(TIMEOUT, cfg.getPropertyCacheTimeout(NON_CACHED));
assertEquals(TIMEOUT, cfg.getPropertyCacheTimeout(CACHED));
Graph graph = open(cfg);
load(graph);
AccumuloVertex a = (AccumuloVertex) graph.getVertex("A");
AccumuloVertex b = (AccumuloVertex) graph.getVertex("B");
AccumuloVertex c = (AccumuloVertex) graph.getVertex("C");
assertEquals(null, a.getProperty(NON_CACHED));
assertEquals(true, b.getProperty(NON_CACHED));
assertEquals(null, c.getProperty(NON_CACHED));
assertEquals(null, a.getProperty(CACHED));
assertEquals(null, b.getProperty(CACHED));
assertEquals(true, c.getProperty(CACHED));
assertEquals(null, a.getPropertyCache().get(NON_CACHED));
assertEquals(true, b.getPropertyCache().get(NON_CACHED));
assertEquals(null, c.getPropertyCache().get(NON_CACHED));
assertEquals(null, a.getPropertyCache().get(CACHED));
assertEquals(null, b.getPropertyCache().get(CACHED));
assertEquals(true, c.getPropertyCache().get(CACHED));
assertEquals(Sets.newHashSet(), a.getPropertyCache().keySet());
assertEquals(Sets.newHashSet(NON_CACHED), b.getPropertyCache().keySet());
assertEquals(Sets.newHashSet(CACHED), c.getPropertyCache().keySet());
a.removeProperty(NON_CACHED);
b.removeProperty(NON_CACHED);
c.removeProperty(NON_CACHED);
a.removeProperty(CACHED);
b.removeProperty(CACHED);
c.removeProperty(CACHED);
assertEquals(null, a.getProperty(NON_CACHED));
assertEquals(null, b.getProperty(NON_CACHED));
assertEquals(null, c.getProperty(NON_CACHED));
assertEquals(null, a.getProperty(CACHED));
assertEquals(null, b.getProperty(CACHED));
assertEquals(null, c.getProperty(CACHED));
assertEquals(null, a.getPropertyCache().get(NON_CACHED));
assertEquals(null, b.getPropertyCache().get(NON_CACHED));
assertEquals(null, c.getPropertyCache().get(NON_CACHED));
assertEquals(null, a.getPropertyCache().get(CACHED));
assertEquals(null, b.getPropertyCache().get(CACHED));
assertEquals(null, c.getPropertyCache().get(CACHED));
assertEquals(Sets.newHashSet(), a.getPropertyCache().keySet());
assertEquals(Sets.newHashSet(), b.getPropertyCache().keySet());
assertEquals(Sets.newHashSet(), c.getPropertyCache().keySet());
graph.shutdown();
}
@Test
public void testPreloadAllProperties() {
AccumuloGraphConfiguration cfg =
AccumuloGraphTestUtils.generateGraphConfig("preloadAllProperties");
cfg.setPropertyCacheTimeout(null, TIMEOUT);
cfg.setPreloadAllProperties(true);
Graph graph = open(cfg);
AccumuloVertex v = (AccumuloVertex) graph.addVertex("V");
v.setProperty(NON_CACHED, true);
v.setProperty(CACHED, true);
v = (AccumuloVertex) graph.getVertex("V");
assertEquals(true, v.getPropertyInMemory(NON_CACHED));
assertEquals(true, v.getPropertyInMemory(CACHED));
graph.shutdown();
}
@Test
public void testPreloadSomeProperties() {
AccumuloGraphConfiguration cfg =
AccumuloGraphTestUtils.generateGraphConfig("preloadSomeProperties");
cfg.setPropertyCacheTimeout(null, TIMEOUT);
cfg.setPreloadedProperties(new String[]{CACHED});
Graph graph = open(cfg);
AccumuloVertex v = (AccumuloVertex) graph.addVertex("V");
v.setProperty(NON_CACHED, true);
v.setProperty(CACHED, true);
v = (AccumuloVertex) graph.getVertex("V");
assertEquals(null, v.getPropertyInMemory(NON_CACHED));
assertEquals(true, v.getPropertyInMemory(CACHED));
graph.shutdown();
}
private static Graph open(AccumuloGraphConfiguration cfg) {
return GraphFactory.open(cfg);
}
private static void load(Graph graph) {
graph.addVertex("A");
graph.addVertex("B").setProperty(NON_CACHED, true);
graph.addVertex("C").setProperty(CACHED, true);
}
}

View File

@@ -4,6 +4,7 @@ import com.tinkerpop.blueprints.Graph;
import com.tinkerpop.blueprints.GraphFactory;
public class ExtendedAccumuloGraphTest extends AccumuloGraphTest {
@Override
public Graph generateGraph(String graphDirectoryName) {
AccumuloGraphConfiguration cfg = AccumuloGraphTestUtils.generateGraphConfig(graphDirectoryName);
@@ -14,4 +15,5 @@ public class ExtendedAccumuloGraphTest extends AccumuloGraphTest {
testGraphName.set(graphDirectoryName);
return GraphFactory.open(cfg.getConfiguration());
}
}

View File

@@ -18,6 +18,8 @@ import static org.junit.Assert.*;
import org.junit.Test;
import edu.jhuapl.tinkerpop.cache.PropertyCache;
/**
* Test the {@link PropertyCache} object.
*/

View File

@@ -1,34 +1,93 @@
Table Structure
---------------
AccumuloGraph Table Schema
==========================
This file documents the structure of backend tables
used for storing elements, properties, etc.
AccumuloGraph uses a number of backing tables for data storage.
This file documents the structure and organization of these
tables. The tables and records are structured in such a way as
to implement the Blueprints operations in terms of efficient Accumulo
operations, i.e., prefix searches, contiguous scans, and
batch operations.
For our purposes, Accumulo entries consist of four fields:
row (R), column family (CF), column qualifier (CQ), and value (V).
For more information on Accumulo's schema and how to
access records efficiently, see the
[Accumulo documentation](https://accumulo.apache.org/1.5/accumulo_user_manual.html).
**Note (29 Dec 2014): This documentation is out of date.**
The tables used by AccumuloGraph are prefixed by the configured graph
name and can be classed as element/property tables, and
index tables. Their structure is discussed below.
## Vertex Table
Elements and properties
-----------------------
Row ID | Column Family | Column Qualifier | Value
---|---|---|---
VertexID | Label Flag | Exists Flag | [empty]
VertexID | INVERTEX | OutVertexID_EdgeID | Edge Label
VertexID | OUTVERTEX | InVertexID_EdgeID | Edge Label
VertexID | Property Key | [empty] | Serialized Value
Vertex and edge information, along with their properties, are stored
in the *graphname*\_vertex and *graphname*\_edge tables respectively.
## Edge Table
First, an entry declares the existence of a vertex in the vertex table.
Row ID | Column Family | Column Qualifier | Value
---|---|---|---
EdgeID|Label Flag|InVertexID_OutVertexID|Edge Label
EdgeID|Property Key|[empty]|Serialized Value
| R | CF | CQ | V |
|---|----|----|---|
| *vertex_id* | `_LABEL_` | `_EXISTS_`| *[empty]* |
## Edge/Vertex Index
Row ID | Column Family | Column Qualifier | Value
---|---|---|---
Serialized Value|Property Key|VertexID/EdgeID|[empty]
A similar entry declares the existence of an edge in the edge table.
## Metadata Table
| R | CF | CQ | V |
|---|----|----|---|
| *edge_id* | `_LABEL_` | *in_vertex_id*`_DELIM_`*out_vertex_id* | *edge_label* |
Row ID | Column Family | Column Qualifier | Value
---|---|---|---
Index Name| Index Class |[empty]|[empty]
When adding an edge, additional entries are stored in the
vertex table for each endpoint of the edge. These facilitate the
`Vertex.getEdge` and `Vertex.getVertex` operations.
| R | CF | CQ | V |
|---|----|----|---|
| *in_vertex_id* | `_IN_EDGE_` | *out_vertex_id*`_DELIM_`*edge_id* | *edge_label* |
| *out_vertex_id* | `_OUT_EDGE_` | *in_vertex_id*`_DELIM_`*edge_id* | *edge_label* |
Finally, vertex and edge properties are stored in their respective
tables. Entry formats are the same for both vertices and edges.
Note that property values are serialized such that their type
can be deduced when deserializing.
| R | CF | CQ | V |
|---|----|----|---|
| *element_id* | *property_key* | *[empty]* | *property_value* |
Indexes
-------
Several tables store index-related information,
including index value tables that store index
property keys and values, and index metadata tables
that store information about what indexes exist
and what properties are indexed.
For `KeyIndexableGraph`, index value tables
include *graphname*\_vertex\_key\_index
and *graphname*\_edge\_key\_index for vertex
and edge properties, respectively.
For `IndexableGraph`, index value tables are
named *graphname*\_index\_*indexname*,
where *indexname* is the index name.
The entry formats in all these tables are the same:
| R | CF | CQ | V |
|---|----|----|---|
| *property_value* | *property_key* | *element_id* | *[empty]* |
Property values are serialized in the same way as above.
In addition to the index value tables, an index metadata table,
*graphname*\_index\_metadata, stores index information. For
`KeyIndexableGraph`, records in this table enumerate the property keys
that are indexed.
| R | CF | CQ | V |
|---|----|----|---|
| *property_key* | `_INDEX_KEY_` | *element_class* | *[empty]* |
For `IndexableGraph`, records enumerate the existing indexes.
| R | CF | CQ | V |
|---|----|----|---|
| *index_name* | `_INDEX_NAME_` | *element_class* | *[empty]* |