180 Commits
0.0.2 ... 0.2.1

Author SHA1 Message Date
Ryan Webb
718258e828 2.1 Release 2015-02-20 12:32:04 -05:00
Ryan Webb
99e01d8608 Added changelog 2015-02-20 12:28:35 -05:00
mikelieberman
a42ae0a521 Merge pull request #108 from JHUAPL/connector-reuse
Added functionality to reuse Connector, and prohibit configuration changes. Addresses #85 - a connector pool is not used, but a single instance is instead.
2015-02-09 17:29:37 -05:00
Michael Lieberman
00056f6513 Added functionality to reuse Connector, and prohibit configuration changes after use 2015-02-09 17:20:48 -05:00
mikelieberman
8df511c2d1 Merge pull request #107 from JHUAPL/preload-all-properties
Preload all properties
2015-02-09 16:17:45 -05:00
Michael Lieberman
ec13b05459 Added "preload all" functionality and unit tests for same 2015-02-09 16:08:10 -05:00
Michael Lieberman
37157a0093 Added unit test for property preloading 2015-02-09 15:48:45 -05:00
Michael Lieberman
438830f5f4 Added setPreloadAllProperties to AccumuloGraphConfiguration 2015-02-09 15:06:36 -05:00
mikelieberman
1018d48382 Merge pull request #106 from JHUAPL/merged-metadata-tables
Merged metadata tables
2015-02-09 13:52:30 -05:00
mikelieberman
30468ed2f3 Updated table structure documentation to match metadata table changes 2015-02-09 13:43:27 -05:00
Michael Lieberman
6faac76ba9 Merge metadata tables into one table 2015-02-09 13:35:06 -05:00
mikelieberman
bb6f9e80a7 Merge pull request #105 from JHUAPL/get-vertices-id-range
Get vertices id range
2015-02-06 18:47:53 -05:00
Michael Lieberman
56929af885 Added documentation and unit test for getVerticesInRange 2015-02-06 18:42:13 -05:00
Michael Lieberman
89976c2c86 Add getVerticesInRange functionality 2015-02-06 18:16:40 -05:00
mikelieberman
816dffe141 Merge pull request #104 from JHUAPL/table-structure-documentation
Table structure documentation
2015-02-06 17:58:13 -05:00
mikelieberman
a7035e259a Updated table structure documentation 2015-02-06 17:55:34 -05:00
Michael Lieberman
f745ff23fb Initial commit of table structure documentation; still needs work 2015-02-06 15:24:07 -05:00
Michael Lieberman
93c9c37936 Replace printStackTrace with exceptions 2015-02-05 17:27:34 -05:00
Michael Lieberman
c08bb5058f FindBugs tweaks 2015-01-28 15:52:58 -05:00
webbrl1
dcea092c40 Fixed High and Medium Coverity bugs. Fixed a few Low bugs 2015-01-28 14:25:11 -05:00
webbrl1
f9bb666e4b Release and updated version 2015-01-28 13:26:09 -05:00
Ryan Webb
834e2cb2aa Merge pull request #99 from JHUAPL/table-wrappers
Table wrappers
2015-01-28 13:03:55 -05:00
Michael Lieberman
f72b1d2f2f Use mutator for adding edge in bulk ingester 2015-01-27 16:38:21 -05:00
Michael Lieberman
5dd5c978a7 Convert PropertyBuilder to use Mutators 2015-01-27 16:26:55 -05:00
Michael Lieberman
bca5bf4474 Use mutator for bulk ingester addProperty 2015-01-27 16:14:13 -05:00
Michael Lieberman
365dc740d7 Bulk ingester add vertex using mutator 2015-01-27 16:08:52 -05:00
Michael Lieberman
d5de29c000 Move createIndex to NamedIndexListTableWrapper 2015-01-27 14:32:55 -05:00
Michael Lieberman
0b2ec50eeb Move createKeyIndex to appropriate wrappers
Some other code cleanup
2015-01-27 14:22:17 -05:00
Michael Lieberman
8a36bc7930 Get rid of graph member from GlobalInstances 2015-01-27 11:44:15 -05:00
Michael Lieberman
38b22df42f Fix AutoIndexTest unit tests and bug fixes 2015-01-27 11:18:12 -05:00
Michael Lieberman
1123268b37 Use TableOperations for isEmpty 2015-01-26 17:17:43 -05:00
Michael Lieberman
be81489672 Clean up unused functionality 2015-01-26 17:07:48 -05:00
Michael Lieberman
37774d60a1 Fix AccumuloVertex.remove, and probably another bug with AccumuloEdge.remove 2015-01-26 16:56:03 -05:00
Michael Lieberman
e2c5af3752 Added delete vertex functionality to VertexTableWrapper 2015-01-26 15:30:37 -05:00
Michael Lieberman
9afc558d95 Move getIndex functionality to NamedIndexListTableWrapper 2015-01-26 12:06:24 -05:00
Michael Lieberman
a2502f5bb1 Move getIndices functionality to NamedIndexListTableWrapper 2015-01-26 11:56:02 -05:00
Michael Lieberman
49a5a77d1d Refactor table wrapper names to avoid confusion 2015-01-26 10:31:00 -05:00
Michael Lieberman
53f7ed4025 Get rid of old non-useful documentation 2015-01-26 09:55:10 -05:00
Michael Lieberman
9edf2ff66e Refactoring 2015-01-23 16:38:29 -05:00
Michael Lieberman
49d4463cc5 Cleanup 2015-01-23 14:44:39 -05:00
Michael Lieberman
37593c13b4 Refactor edge.remove to use existing rather than new functionality 2015-01-23 14:38:23 -05:00
Michael Lieberman
7bf543e502 Move removeEdge functionality into AccumuloEdge
Other cleanup
2015-01-23 14:22:20 -05:00
Michael Lieberman
43a3293ef0 Clean up removeEdge 2015-01-23 14:11:54 -05:00
Michael Lieberman
6fffd094f3 Remove 'cacheProperty' methods from AccumuloElement 2015-01-23 13:28:11 -05:00
Michael Lieberman
b60f855a5e Connect index-clearing code to AccumuloIndex
Other cleanup
2015-01-21 17:51:33 -05:00
Michael Lieberman
832d597f0a Add removeElementFromIndex to IndexTableWrapper 2015-01-21 17:37:36 -05:00
Michael Lieberman
7fb9cce693 Cleanup 2015-01-21 13:47:44 -05:00
Michael Lieberman
128b40346a Cleanup 2015-01-21 13:42:37 -05:00
Michael Lieberman
eacf44c755 Refactor AccumuloIndex put into IndexTableWrapper 2015-01-21 13:37:17 -05:00
Michael Lieberman
e59c590eae Move get functionality out of AccumuloIndex to IndexTableWrapper 2015-01-21 13:24:00 -05:00
Michael Lieberman
2dd9e66cf2 Use indexWrapper for AccumuloIndex.remove 2015-01-21 12:03:17 -05:00
Michael Lieberman
3897998fca Use index parser in AccumuloIndex 2015-01-21 12:01:37 -05:00
Michael Lieberman
a9d8ff7570 Added NamedIndexTableWrapper
Cleanup
2015-01-21 11:54:03 -05:00
Michael Lieberman
7e79174b1d Cleanup 2015-01-21 11:17:46 -05:00
Michael Lieberman
df0824f7de Cleanup 2015-01-20 17:04:35 -05:00
Michael Lieberman
864cee9312 Cleanup and documentation 2015-01-20 16:42:43 -05:00
Michael Lieberman
2544ac0c30 Move index-based getEdges to EdgeIndexTableWrapper 2015-01-20 16:36:07 -05:00
Michael Lieberman
16b7284f08 Move index-based getVertices to VertexIndexTableWrapper 2015-01-20 16:26:33 -05:00
Michael Lieberman
f0bd6b8cd6 Add index entry parsers 2015-01-20 16:04:23 -05:00
Michael Lieberman
379099adc2 Cleanup 2015-01-20 15:32:11 -05:00
Michael Lieberman
f0a23e3848 Move remote property from index functionality to IndexTableWrapper 2015-01-20 15:31:42 -05:00
Michael Lieberman
ad6ef0c2f1 Warnings cleanup 2015-01-20 15:27:20 -05:00
Michael Lieberman
5632417f30 Add IndexValueMutator mutator 2015-01-20 15:23:57 -05:00
Michael Lieberman
c17693968c Cleanup 2015-01-20 14:43:09 -05:00
Michael Lieberman
c66d86d123 Move setPropertyForIndex to IndexTableWrapper 2015-01-20 14:41:15 -05:00
Michael Lieberman
36ca8641ef Cleanup 2015-01-20 14:21:29 -05:00
Michael Lieberman
5032b72803 Add property index tables to globals 2015-01-20 13:52:26 -05:00
Michael Lieberman
c90a379e2b Rename tables to make their purpose clearer and other cleanup 2015-01-20 13:46:16 -05:00
Michael Lieberman
9832b92d7b Move getIndexedKeys to KeyMetadataTableWrapper
Also some other cleanup
2015-01-19 19:04:54 -05:00
Michael Lieberman
712420c2c5 Use IndexMetadataTableWrapper for index metadata
Other cleanup
2015-01-19 18:25:28 -05:00
Michael Lieberman
2c08dc149e Use same table format for key metadata and index metadata 2015-01-19 18:09:36 -05:00
Michael Lieberman
484e78f080 Documentation and cleanup 2015-01-19 17:57:13 -05:00
Michael Lieberman
2b3f07688d Several updates
Created key index metadata wrapper and moved some functionality there
Changed Mutator to an interface rather than abstract class
2015-01-19 17:42:37 -05:00
Michael Lieberman
ff17e6a5af Tweaks and cleanup 2015-01-19 15:16:27 -05:00
Michael Lieberman
72c9685450 Tweaks 2015-01-19 15:13:58 -05:00
Michael Lieberman
42afab5968 Partly move getEdges into EdgeTableWrapper 2015-01-19 15:12:52 -05:00
Michael Lieberman
332df1fea4 Clean up / reorganize non-useful / duplicate code 2015-01-19 15:06:37 -05:00
Michael Lieberman
6196dc7d8f Move key/value getVertices into VertexTableWrapper 2015-01-19 14:23:20 -05:00
Michael Lieberman
98a24bd62f Abstracted entry parser for vertex, edge and property parsers 2015-01-05 19:43:55 -05:00
Michael Lieberman
c9b7473df0 Remove cruft 2015-01-05 19:33:59 -05:00
Michael Lieberman
886c394afb Hide debug message 2015-01-05 19:33:15 -05:00
Michael Lieberman
8b7f8b7e9d Add log4j.properties 2015-01-05 19:33:03 -05:00
Michael Lieberman
25eb8e9979 Move cache items to their own package 2015-01-05 19:30:36 -05:00
Michael Lieberman
33042acedc Fix header licenses 2015-01-05 19:28:38 -05:00
Michael Lieberman
2465f65482 Merge branch 'master' into table-wrappers 2015-01-05 19:24:09 -05:00
Michael Lieberman
899ed892fc Move getEdges to EdgeTableWrapper
Make getVertices and getEdges cache their results
Remove confusing AccumuloEdge constructors and other methods
Added some debugging method for the backing tables
Bug fixes in unit tests
2015-01-05 19:23:57 -05:00
Michael Lieberman
ad7d405339 Update to next snapshot version 2015-01-05 16:58:06 -05:00
Michael Lieberman
e68f396dfe Update README version 2015-01-05 16:48:25 -05:00
Michael Lieberman
8afc46ce8b Update version number 2015-01-05 16:29:08 -05:00
Michael Lieberman
7c8996ba8b Fix licenses in headers 2015-01-05 15:41:37 -05:00
Michael Lieberman
37b4c8adc9 Refactor conversion from entries to elements as ElementParser and other classes 2015-01-05 15:10:06 -05:00
Michael Lieberman
76b0007434 Move getVertices() into VertexTableWrapper
Other cleanup
2015-01-05 14:33:48 -05:00
Michael Lieberman
3a08194ba3 Merge branch 'property-updates' into table-wrappers
Conflicts:
	src/main/java/edu/jhuapl/tinkerpop/AccumuloElement.java
	src/main/java/edu/jhuapl/tinkerpop/AccumuloGraph.java
2015-01-05 14:32:19 -05:00
Michael Lieberman
65b8d43d42 Cruft cleanup 2014-12-31 12:57:04 -05:00
Michael Lieberman
8b8fc0c7e6 Moved removeProperty functionality into AccumuloElement 2014-12-31 12:03:43 -05:00
Michael Lieberman
fec6fdc522 Partly move setProperty functionality into AccumuloElement 2014-12-31 11:51:53 -05:00
Michael Lieberman
439dc7ae93 Clean up test framework configuration 2014-12-31 01:39:51 -05:00
Michael Lieberman
ac68e334a6 Use a single writer in the table wrappers 2014-12-31 01:35:28 -05:00
Michael Lieberman
f12e78d836 Property cache updates
Added property caching unit test
Bug fix in property caching
Cruft cleanup and deprecate AccumuloGraph methods
2014-12-31 01:31:29 -05:00
Michael Lieberman
2d7a0d4c22 Property updates
Moved removeProperty to AccumuloElement
Added property cache unit tests
Moved flushing to GlobalInstances
Cruft cleanup

NOTE: Unit tests failing due to what appears to be race condition
2014-12-30 19:21:25 -05:00
Michael Lieberman
06bb292b42 Move getProperty and getPropertyKeys inside AccumuloElement
Other cruft cleanup
2014-12-30 16:56:20 -05:00
Michael Lieberman
6b85e54628 Move getVertices() into VertexTableWrapper 2014-12-30 15:46:20 -05:00
Michael Lieberman
ed77c11b1c Collect element caches into single object 2014-12-30 15:42:10 -05:00
Michael Lieberman
a61fc8ba20 Refactor getEdges to use Vertex object 2014-12-30 15:10:21 -05:00
Michael Lieberman
6613bcd569 Added edge endpoint add/delete
Added operation to batch delete elements based on range
Refactored add/delete operations into final classes
Updated documentation
2014-12-30 15:02:44 -05:00
Michael Lieberman
80ddbd3127 Initial commits for effort to genericize mutators based on graph operations 2014-12-30 14:38:15 -05:00
Michael Lieberman
c1429b3bf3 Several changes
Refactoring of wrappers to use GlobalInstances object
Added VertexTableWrapper and EdgeTableWrapper to GlobalInstances
Started moving vertex.getEdges() to VertexTableWrapper
2014-12-29 18:21:38 -05:00
Michael Lieberman
aa860b3441 Refactor property operations to use GlobalInstances object 2014-12-29 17:39:18 -05:00
Michael Lieberman
1c27add8e5 Refactor VertexTableWrapper and EdgeTableWrapper operations to use element objects 2014-12-29 17:31:30 -05:00
Michael Lieberman
9c2e07c395 Pass around a "GlobalInstances" object to simplify passing globals relevant to multiple AccumuloGraph objects 2014-12-29 17:26:53 -05:00
Michael Lieberman
bcc4af3599 Rename configuration methods for more clarity 2014-12-29 16:22:26 -05:00
Michael Lieberman
85544dea77 Merge branch 'master' into table-wrappers 2014-12-29 15:52:12 -05:00
Michael Lieberman
75df33d51c Update to SNAPSHOT version (for now) 2014-12-29 15:26:44 -05:00
mikelieberman
e126a52c32 Merge pull request #93 from JHUAPL/readme-update
Readme update
2014-12-29 15:22:02 -05:00
Michael Lieberman
f0d3898c2d More README.md updates 2014-12-29 15:11:35 -05:00
Michael Lieberman
ee45dc4ade Initial commit for table structure documentation 2014-12-29 14:28:37 -05:00
Michael Lieberman
f90290bb3d Let's use AccumuloGraphException instead of RuntimeException 2014-12-29 14:13:36 -05:00
Michael Lieberman
8c6c02dffb Lots of updates to README.md 2014-12-29 14:13:00 -05:00
mikelieberman
e2dccb15c2 Merge pull request #91 from JHUAPL/unit-test-updates
Unit test updates
2014-12-29 14:12:29 -05:00
Michael Lieberman
7a829d5c34 setCreate by default for DistributedInstanceTest 2014-12-29 13:42:28 -05:00
Michael Lieberman
0f854ec2b7 Separate out Features creation 2014-12-29 12:33:04 -05:00
Michael Lieberman
99c19669f9 Unit test updates
Initial commits for Mock/Distributed instance tests
New test for valid graph names
Update configuration validation
2014-12-29 12:20:29 -05:00
Michael Lieberman
f94bb2e9f4 Make AccumuloGraphConfiguration extend AbstractConfiguration for simpler usage 2014-12-29 11:03:55 -05:00
Michael Lieberman
ef978884bc By default, set the create option for mock graphs 2014-12-29 10:21:25 -05:00
Michael Lieberman
7ffd181dbe Bug fix with adding elements with non-String ids
Added corresponding unit test
2014-12-29 10:20:32 -05:00
Michael Lieberman
2ffdb81fe3 Refactoring property parsing to separate method in anticipation of vertex/edge-specific entry parsing 2014-12-24 16:52:18 -05:00
Michael Lieberman
c29983966b Add cacheAllProperties utility method 2014-12-24 16:38:59 -05:00
Michael Lieberman
bfba0ed665 Use ElementTableWrapper functionality for preloading edge properties 2014-12-24 16:08:15 -05:00
Michael Lieberman
a3181eac62 Move vertex property loading to ElementTableWrapper 2014-12-24 16:02:00 -05:00
Michael Lieberman
7d2d9bb45a Merge branch 'master' into table-wrappers 2014-12-24 15:26:02 -05:00
Michael Lieberman
bed037caa2 Bug fix in InputFormatsTest
Was reusing the same fake Hadoop table name and causing problems with
existence checks
2014-12-24 15:24:04 -05:00
Michael Lieberman
be75b8fc1f Existence check fixes
Added unit tests for skip existence checks
Bug fix for existence check in addVertex
2014-12-24 14:59:20 -05:00
Michael Lieberman
fc09015445 Merge branch 'master' into table-wrappers 2014-12-24 13:30:02 -05:00
mikelieberman
805368972f Merge pull request #90 from JHUAPL/cache-updates
Property cache updates
2014-12-24 13:29:01 -05:00
Michael Lieberman
cae1124562 Merge branch 'cache-updates' into table-wrappers
Conflicts:
	src/main/java/edu/jhuapl/tinkerpop/AccumuloElement.java
	src/main/java/edu/jhuapl/tinkerpop/AccumuloGraph.java
2014-12-24 13:07:30 -05:00
Michael Lieberman
101c549f76 Make element cache unit tests a bit faster 2014-12-24 13:02:19 -05:00
Michael Lieberman
fc827aa0ca Property cache refactoring
Added PropertyCache object to consolidate property caching operations
Refactored element objects to use it
Added unit tests
2014-12-24 13:00:51 -05:00
Michael Lieberman
aa4897ca27 Merge branch 'master' into table-wrappers
Conflicts:
	src/main/java/edu/jhuapl/tinkerpop/AccumuloGraph.java
2014-12-24 11:43:26 -05:00
mikelieberman
8b120f66ed Merge pull request #88 from JHUAPL/element-cache-updates
Element cache updates
2014-12-24 11:41:34 -05:00
Michael Lieberman
9117cab1ae Update element caching to use Guava cache implementation 2014-12-24 11:35:20 -05:00
Michael Lieberman
0526fb68e4 Update configuration object for better cache parameters 2014-12-24 10:55:39 -05:00
Michael Lieberman
5affcea6e7 Merge branch 'master' into table-wrappers 2014-12-23 19:15:51 -05:00
Michael Lieberman
1acc82af57 Revert "Make the vertex/edge caches have no default timeout, as is mentioned in documentation"
This reverts commit 1590fe05fd.

It is not clear that existing cache implementation works after changing this default.
2014-12-23 19:15:21 -05:00
Michael Lieberman
1590fe05fd Make the vertex/edge caches have no default timeout, as is mentioned in documentation 2014-12-23 19:12:13 -05:00
Michael Lieberman
33cc53c00e Move edge writing methods to EdgeTableWrapper / VertexTableWrapper
Some more refactoring and cleanup
2014-12-23 19:01:00 -05:00
Michael Lieberman
0b8ab3361a Move vertex writing to VertexTableWrapper 2014-12-23 18:32:56 -05:00
Michael Lieberman
b991d84a28 Remove unnecessary parent argument to ScannerIterable 2014-12-23 17:53:06 -05:00
Michael Lieberman
bd7cd409d2 Simplification / abstraction of table wrappers 2014-12-23 17:32:44 -05:00
Michael Lieberman
b7d9d3f340 Initial commits of index wrappers
Remove unused methods from AccumuloGraphConfiguration
2014-12-23 17:27:38 -05:00
Michael Lieberman
0bf11f444e Create BaseTableWrapper and refactor ElementTableWrapper
Also added some Override annotations and other cleanup
2014-12-23 17:09:39 -05:00
Michael Lieberman
5d7442e024 Rename "tableoperations" to "tablewrapper" classes
Move TableWrapper classes to "tables" package
2014-12-23 16:53:45 -05:00
Michael Lieberman
670e4b5c85 Move setProperty functionality (not including indexes) to ElementTableOperations
Remove unused method in AccumuloGraph
Also, use AccumuloGraphException in more places
2014-12-23 16:42:07 -05:00
Michael Lieberman
2db2960b67 Move removeProperty functionality (not including indexes) to ElementTableOperations 2014-12-23 16:20:35 -05:00
Michael Lieberman
f378931c3a Move readPropertyKeys to ElementTableOperations 2014-12-23 15:53:05 -05:00
Michael Lieberman
b749943cc9 Move getProperty in AccumuloGraph to ElementTableOperations 2014-12-23 15:42:56 -05:00
Michael Lieberman
d20da99ad3 Initial commits for ElementTableOperations and subclasses 2014-12-23 15:39:16 -05:00
Michael Lieberman
6eba6cc502 More documentation cleanup and cruft removal 2014-12-23 15:01:58 -05:00
Michael Lieberman
b5d5c6f5bc Delete unused cruft classes 2014-12-23 13:55:27 -05:00
Michael Lieberman
2a74321b5a More source and typo cleanup 2014-12-23 13:51:29 -05:00
Michael Lieberman
1b58eb42e8 Add override annotations in AccumuloGraph 2014-12-23 11:24:09 -05:00
Michael Lieberman
100c46aee9 Re-added some old (deprecated) configuration method names 2014-12-23 10:21:10 -05:00
mikelieberman
f2406b8f07 Merge pull request #81 from JHUAPL/configuration-cleanup
Configuration cleanup
2014-12-22 17:18:41 -05:00
Michael Lieberman
7f8c98edc6 Add print option for AccumuloGraphConfiguration
JUnit refactoring
2014-12-15 19:15:12 -05:00
Michael Lieberman
1309512331 Lots of configuration cleanup and refactoring 2014-12-15 17:45:33 -05:00
liebemd1
c42c06f6e2 Add null check on closing AccumuloBulkIngester 2014-11-17 12:54:38 -05:00
Ryan Webb
dd98652d92 Updated the pom to have hadoop jars provided.
Fixed configuration for rexster use
2014-11-10 09:01:58 -05:00
Ryan Webb
de42c76fa5 Merge pull request #77 from JHUAPL/testingDependencies
Updated pom to remove dependencies that mess with webservers
2014-11-07 14:46:49 -05:00
webbrl1
e1d359ab83 Updated pom to remove dependencies that mess with webservers 2014-11-07 14:39:04 -05:00
Michael Lieberman
b52b441dfd Add fatjar profile 2014-10-31 15:12:13 -04:00
Michael Lieberman
db35a8f620 Error checking on AccumuloGraphConfiguration initialization 2014-10-27 11:08:20 -04:00
Michael Lieberman
0d9eba2211 Documentation 2014-10-27 10:58:20 -04:00
Michael Lieberman
955177b46e In AccumuloByteSerializer, return null if null value 2014-10-24 17:08:36 -04:00
Michael Lieberman
d699dbee8f Documentation fixes 2014-10-24 16:01:32 -04:00
Michael Lieberman
60e455a4ba Added Javadocs for AccumuloGraphConfiguration properties 2014-10-24 14:55:55 -04:00
Ryan Webb
289f65999d Added Javadoc Profile 2014-10-24 12:48:50 -04:00
Michael Lieberman
507ffa180c Documentation 2014-10-17 13:08:48 -04:00
Michael Lieberman
1b10773c6c Make method names consistent with naming scheme 2014-10-17 12:06:04 -04:00
Michael Lieberman
83e227e760 Fix spelling 2014-10-17 11:59:20 -04:00
Michael Lieberman
0b23289d07 Fix a few warnings 2014-10-16 11:11:48 -04:00
Ryan Webb
abd1771803 Updating to next snapshot 2014-10-08 15:16:17 -04:00
72 changed files with 6138 additions and 2608 deletions

292
README.md
View File

@@ -4,140 +4,210 @@ AccumuloGraph
This is an implementation of the [TinkerPop Blueprints](http://tinkerpop.com)
2.6 API using [Apache Accumulo](http://apache.accumulo.com) as the backend.
This implementation provides easy to use, easy to write, and easy to read
access to an arbitrarily large graph that is stored in Accumulo.
We implement the following Blueprints interfaces:
<br>1. Graph
<br>2. KeyIndexableGraph
<br>3. IndexableGraph
Please feel free to submit issues for any bugs you find or features you want.
We are open to pull requests from your forks also.
This combines the many benefits and flexibility of Blueprints
with the scalability and performance of Accumulo.
##Usage
In addition to the basic Blueprints functionality, we provide a number
of enhanced features, including:
* Indexing implementations via `IndexableGraph` and `KeyIndexableGraph`
* Support for mock, mini, and distributed instances of Accumulo
* Numerous performance tweaks and configuration parameters
* Support for high speed ingest
* Hadoop integration
Feel free to contact us with bugs, suggestions, pull requests,
or simply how you are leveraging AccumuloGraph in your own work.
## Getting Started
First, include AccumuloGraph as a Maven dependency. Releases are deployed
to Maven Central.
The releases are currently stored in Maven Central.
```xml
<dependency>
<groupId>edu.jhuapl.tinkerpop</groupId>
<artifactId>blueprints-accumulo-graph</artifactId>
<version>0.0.1</version>
<version>0.2.0</version>
</dependency>
```
For non-Maven users, the binaries can be found in the releases section in this
For non-Maven users, the binary jars can be found in the releases section in this
GitHub repository, or you can get them from Maven Central.
##Code Examples
###Creating a new or connecting to an existing distributed graph
Creating an `AccumuloGraph` involves setting a few parameters in an
`AccumuloGraphConfiguration` object, and opening the graph.
The defaults are sensible for using an Accumulo cluster.
We provide some simple examples below. Javadocs for
`AccumuloGraphConfiguration` explain all the other parameters
in more detail.
First, to instantiate an in-memory graph:
```java
Configuration cfg = new AccumuloGraphConfiguration()
.setInstanceName("accumulo").setUser("user").setZookeeperHosts("zk1")
.setPassword("password".getBytes()).setGraphName("myGraph");
Graph graph = GraphFactory.open(cfg.getConfiguration());
.setInstanceType(InstanceType.Mock)
.setGraphName("graph");
return GraphFactory.open(cfg);
```
###Creating a new Mock Graph
Setting the instance type to mock allows for in-memory processing with a MockAccumulo instance.<br>
There is also support for Mini Accumulo.
This creates a "Mock" instance which holds the graph in memory.
You can now use all the Blueprints and AccumuloGraph-specific functionality
with this in-memory graph. This is useful for getting familiar
with AccumuloGraph's functionality, or for testing or prototyping
purposes.
To use an actual Accumulo cluster, use the following:
```java
Configuration cfg = new AccumuloGraphConfiguration().setInstanceType(InstanceType.Mock)
.setGraphName("myGraph");
Graph graph = GraphFactory.open(cfg);
Configuration cfg = new AccumuloGraphConfiguration()
.setInstanceType(InstanceType.Distributed)
.setZooKeeperHosts("zookeeper-host")
.setInstanceName("instance-name")
.setUser("user").setPassword("password")
.setGraphName("graph")
.setCreate(true);
return GraphFactory.open(cfg);
```
###Accessing a graph
This directs AccumuloGraph to use a "Distributed" Accumulo
instance, and sets the appropriate ZooKeeper parameters,
instance name, and authentication information, which correspond
to the usual Accumulo connection settings. The graph name is
used to create several backing tables in Accumulo, and the
`setCreate` option tells AccumuloGraph to create the backing
tables if they don't already exist.
AccumuloGraph also has limited support for a "Mini" instance
of Accumulo.
## Improving Performance
This section describes various configuration parameters that
greatly enhance AccumuloGraph's performance. Brief descriptions
of each option are provided here, but refer to the
`AccumuloGraphConfiguration` Javadoc for fuller explanations.
### Disable consistency checks
The Blueprints API specifies a number of consistency checks for
various operations, and requires errors if they fail. Some examples
of invalid operations include adding a vertex with the same id as an
existing vertex, adding edges between nonexistent vertices,
and setting properties on nonexistent elements.
Unfortunately, checking the above constraints for an
Accumulo installation entails significant performance issues,
since these require extra traffic to Accumulo using inefficient
non-batched access patterns.
To remedy these performance issues, AccumuloGraph exposes
several options to disable various of the above checks.
These include:
* `setAutoFlush` - to disable automatically flushing
changes to the backing Accumulo tables
* `setSkipExistenceChecks` - to disable element
existence checks, avoiding trips to the Accumulo cluster
* `setIndexableGraphDisabled` - to disable
indexing functionality, which improves performance
of element removal
### Tweak Accumulo performance parameters
Accumulo itself features a number of performance-related parameters,
and we allow configuration of these. Generally, these relate to
write buffer sizes, multithreading, etc. The settings include:
* `setMaxWriteLatency` - max time prior to flushing
element write buffer
* `setMaxWriteMemory` - max size for element write buffer
* `setMaxWriteThreads` - max threads used for element writing
* `setMaxWriteTimeout` - max time to wait before failing
element buffer writes
* `setQueryThreads` - number of query threads to use
for fetching elements, properties etc.
### Enable edge and property preloading
As a performance tweak, AccumuloGraph performs lazy loading of
properties and edges. This means that an operation such as
`getVertex` does not by default populate the returned
vertex object with the associated vertex's properties
and edges. Instead, they are initialized only when requested via
`getProperty`, `getEdges`, etc. These are useful
for use cases where you won't be accessing many of these
properties. However, if certain properties or edges will
be accessed frequently, you can set options for preloading
these specific properties and edges, which will be more
efficient than on-the-fly loading. These options include:
* `setPreloadedProperties` - set property keys
to be preloaded
* `setPreloadedEdgeLabels` - set edges to be
preloaded based on their labels
### Enable caching
AccumuloGraph contains a number of caching options
that mitigate the need for Accumulo traffic for recently-accessed
elements. The following options control caching:
* `setVertexCacheParams` - size and expiry for vertex cache
* `setEdgeCacheParams` - size and expiry for edge cache
* `setPropertyCacheTimeout` - property expiry time,
which can be specified globally and/or for individual properties
## High Speed Ingest
One of Accumulo's key advantages is its ability for high-speed ingest
of huge amounts of data. To leverage this ability, we provide
an additional `AccumuloBulkIngester` class that
exchanges consistency guarantees for high speed ingest.
The following is an example of how to use the bulk ingester to
ingest a simple graph:
```java
Vertex v1 = graph.addVertex("1");
v1.setProperty("name", "Alice");
Vertex v2 = graph.addVertex("2");
v2.setProperty("name", "Bob");
Edge e1 = graph.addEdge("E1", v1, v2, "knows");
e1.setProperty("since", new Date());
```
###Creating indexes
```java
((KeyIndexableGraph)graph)
.createKeyIndex("name", Vertex.class);
AccumuloGraphConfiguration cfg = ...;
AccumuloBulkIngester ingester = new AccumuloBulkIngester(cfg);
// Add a vertex.
ingester.addVertex("A").finish();
// Add another vertex with properties.
ingester.addVertex("B")
.add("P1", "V1").add("P2", "V2")
.finish();
// Add an edge.
ingester.addEdge("A", "B", "edge").finish();
// Shutdown and compact tables.
ingester.shutdown(true);
```
###MapReduce Integration
####In the tool
See the Javadocs for more details.
Note that you are responsible for ensuring that data is entered
in a consistent way, or the resulting graph will
have undefined behavior.
## Hadoop Integration
AccumuloGraph features Hadoop integration via custom input and output
format implementations. `VertexInputFormat` and `EdgeInputFormat`
allow vertex and edge inputs to mappers, respectively. Use as follows:
```java
AccumuloConfiguration cfg = new AccumuloGraphConfiguration()
.setInstanceName("accumulo").setZookeeperHosts("zk1").setUser("root")
.setPassword("secret".getBytes()).setGraphName("myGraph");
AccumuloGraphConfiguration cfg = ...;
// For vertices:
Job j = new Job();
j.setInputFormatClass(VertexInputFormat.class);
VertexInputFormat.setAccumuloGraphConfiguration(j,
cfg.getConfiguration());
VertexInputFormat.setAccumuloGraphConfiguration(j, cfg);
// For edges:
Job j = new Job();
j.setInputFormatClass(EdgeInputFormat.class);
EdgeInputFormat.setAccumuloGraphConfiguration(j, cfg);
```
####In the mapper
`ElementOutputFormat` allows writing to an AccumuloGraph from
reducers. Use as follows:
```java
public void map(Text k, Vertex v, Context c) {
System.out.println(v.getId().toString());
}
```
##Table Design
###Vertex Table
Row ID | Column Family | Column Qualifier | Value
---|---|---|---
VertexID | Label Flag | Exists Flag | [empty]
VertexID | INVERTEX | OutVertexID_EdgeID | Edge Label
VertexID | OUTVERTEX | InVertexID_EdgeID | Edge Label
VertexID | Property Key | [empty] | Serialized Value
###Edge Table
Row ID | Column Family | Column Qualifier | Value
---|---|---|---
EdgeID|Label Flag|InVertexID_OutVertexID|Edge Label
EdgeID|Property Key|[empty]|Serialized Value
###Edge/Vertex Index
Row ID | Column Family | Column Qualifier | Value
---|---|---|---
Serialized Value|Property Key|VertexID/EdgeID|[empty]
###Metadata Table
Row ID | Column Family | Column Qualifier | Value
---|---|---|---
Index Name| Index Class |[empty]|[empty]
##Advanced Configuration
###Graph Configuration
- setGraphName(String name)
- setCreate(boolean create) - Sets if the backing graph tables should be created if they do not exist.
- setClear(boolean clear) - Sets if the backing graph tables should be reset if they exist.
- autoFlush(boolean autoFlush) - Sets if each graph element and property change will be flushed to the server.
- skipExistenceChecks(boolean skip) - Sets if you want to skip existance checks when creating graph elemenets.
- setAutoIndex(boolean ison) - Turns on/off automatic indexing.
###Accumulo Control
- setUser(String user) - Sets the user to use when connecting to Accumulo
- setPassword(byte[] password | String password) - Sets the password to use when connecting to Accumulo
- setZookeeperHosts(String zookeeperHosts) - Sets the Zookeepers to connect to.
- setInstanceName(String instance) - Sets the Instance name to use when connecting to Zookeeper
- setInstanceType(InstanceType type) - Sets the type of Instance to use : Distrubuted, Mini, or Mock. Defaults to Distrubuted
- setQueryThreads(int threads) - Specifies the number of threads to use in scanners. Defaults to 3
- setMaxWriteLatency(long latency) - Sets the latency to be used for all writes to Accumulo
- setMaxWriteTimeout(long timeout) - Sets the timeout to be used for all writes to Accumulo
- setMaxWriteMemory(long mem) - Sets the memory buffer to be used for all writes to Accumulo
- setMaxWriteThreads(int threads) - Sets the number of threads to be used for all writes to Accumulo
- setAuthorizations(Authorizations auths) - Sets the authorizations to use when accessing the graph
- setColumnVisibility(ColumnVisibility colVis) - TODO
- setSplits(String splits | String[] splits) - Sets the splits to use when creating tables. Can be a space sperated list or an array of splits
- setMiniClusterTempDir(String dir) - Sets directory to use as the temp directory for the Mini cluster
###Caching
- setLruMaxCapacity(int max) - TODO
- setVertexCacheTimeout(int millis) - Sets the vertex cache timeout. A value <=0 clears the value
- setEdgeCacheTimeout(int millis) - Sets the edge cache timeout. A value <=0 clears the value
###Preloading
- setPropertyCacheTimeout(int millis) - Sets the element property cache timeout. A value <=0 clears the value
- setPreloadedProperties(String[] propertyKeys) - Sets the property keys that should be preloaded. Requiers a positive timout.
- setPreloadedEdgeLabels(String[] edgeLabels) - TODO
AccumuloGraphConfiguration cfg = ...;
Job j = new Job();
j.setOutputFormatClass(ElementOutputFormat.class);
ElementOutputFormat.setAccumuloGraphConfiguration(j, cfg);
```

6
changelog Normal file
View File

@@ -0,0 +1,6 @@
[2.1]
Change Log started
--109,103 Merged Metadata tables
--97 Added range queries
--95 Table structure documentation
--79 All flag for preloading

84
pom.xml
View File

@@ -3,7 +3,7 @@
<modelVersion>4.0.0</modelVersion>
<groupId>edu.jhuapl.tinkerpop</groupId>
<artifactId>blueprints-accumulo-graph</artifactId>
<version>0.0.2</version>
<version>0.2.1</version>
<name>blueprints-accumulo-graph</name>
<description>An Accumulo-backed implementation of the Tinkerpop Blueprints graph API.</description>
<packaging>jar</packaging>
@@ -14,8 +14,6 @@
<url>http://www.apache.org/licenses/LICENSE-2.0.txt</url>
<distribution>repo</distribution>
</license>
</licenses>
<scm>
@@ -43,7 +41,6 @@
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.accumulo</groupId>
@@ -70,24 +67,30 @@
<artifactId>commons-collections</artifactId>
<version>3.2.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.2.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.tinkerpop.rexster</groupId>
<artifactId>rexster-server</artifactId>
<version>2.6.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.tinkerpop.blueprints</groupId>
<artifactId>blueprints-test</artifactId>
<version>2.6.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-minicluster</artifactId>
<version>2.2.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
@@ -100,12 +103,6 @@
<version>3.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.tinkerpop.rexster</groupId>
<artifactId>rexster-server</artifactId>
<version>2.6.0</version>
</dependency>
</dependencies>
<distributionManagement>
@@ -167,6 +164,59 @@
</plugins>
</build>
</profile>
<profile>
<id>javadoc</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
<version>2.2.1</version>
<executions>
<execution>
<id>attach-sources</id>
<goals>
<goal>jar-no-fork</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<version>2.9.1</version>
<executions>
<execution>
<id>attach-javadocs</id>
<goals>
<goal>jar</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
<profile>
<id>fatjar</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
<build>
@@ -182,8 +232,6 @@
<autoReleaseAfterClose>true</autoReleaseAfterClose>
</configuration>
</plugin>
</plugins>
</build>
</project>

View File

@@ -32,11 +32,20 @@ import com.tinkerpop.blueprints.Edge;
import com.tinkerpop.blueprints.GraphFactory;
import com.tinkerpop.blueprints.Vertex;
import edu.jhuapl.tinkerpop.mutator.Mutators;
import edu.jhuapl.tinkerpop.mutator.edge.EdgeEndpointsMutator;
import edu.jhuapl.tinkerpop.mutator.edge.EdgeMutator;
import edu.jhuapl.tinkerpop.mutator.property.WritePropertyMutator;
import edu.jhuapl.tinkerpop.mutator.vertex.AddVertexMutator;
/**
* This class providers high-speed ingest into an AccumuloGraph instance in exchange for consistency guarantees. That is, users of this class must ensure
* (outside of this class) that data is entered in a consistent way or the behavior or the resulting AccumuloGraph is undefined. For example, users are required
* to ensure that a vertex ID provided as the source or destination of an edge exists (or will exist by the end of the ingest process). Likewise, it is the
* This class provides high-speed ingest into an {@link AccumuloGraph} instance
* in exchange for consistency guarantees. That is, users of this class must ensure
* (outside of this class) that data is entered in a consistent way or the behavior
* or the resulting AccumuloGraph is undefined. For example, users are required
* to ensure that a vertex ID provided as the source or destination of an edge exists
* (or will exist by the end of the ingest process). Likewise, it is the
* user's responsibility to ensure vertex and edge IDs provided for properties (will) exist.
* <P>
* TODO define the properties that will be used (vs. those that are ignored) from the provided AccumuloGraphConfiguration.
@@ -88,8 +97,8 @@ public final class AccumuloBulkIngester {
AccumuloGraphUtils.handleCreateAndClear(config);
mtbw = connector.createMultiTableBatchWriter(config.getBatchWriterConfig());
vertexWriter = mtbw.getBatchWriter(config.getVertexTable());
edgeWriter = mtbw.getBatchWriter(config.getEdgeTable());
vertexWriter = mtbw.getBatchWriter(config.getVertexTableName());
edgeWriter = mtbw.getBatchWriter(config.getEdgeTableName());
}
/**
@@ -105,9 +114,7 @@ public final class AccumuloBulkIngester {
* @throws MutationsRejectedException
*/
public PropertyBuilder addVertex(String id) throws MutationsRejectedException {
Mutation m = new Mutation(id);
m.put(AccumuloGraph.LABEL, AccumuloGraph.EXISTS, AccumuloGraph.EMPTY);
vertexWriter.addMutation(m);
Mutators.apply(vertexWriter, new AddVertexMutator(id));
return new PropertyBuilder(vertexWriter, id);
}
@@ -145,8 +152,7 @@ public final class AccumuloBulkIngester {
* @throws MutationsRejectedException
*/
public PropertyBuilder addEdge(String src, String dest, String label) throws MutationsRejectedException {
String eid = UUID.randomUUID().toString();
return addEdge(eid, src, dest, label);
return addEdge(UUID.randomUUID().toString(), src, dest, label);
}
/**
@@ -165,16 +171,8 @@ public final class AccumuloBulkIngester {
* @throws MutationsRejectedException
*/
public PropertyBuilder addEdge(String id, String src, String dest, String label) throws MutationsRejectedException {
Mutation m = new Mutation(id);
m.put(AccumuloGraph.LABEL, (dest + "_" + src).getBytes(), AccumuloByteSerializer.serialize(label));
edgeWriter.addMutation(m);
m = new Mutation(dest);
m.put(AccumuloGraph.INEDGE, (src + AccumuloGraph.IDDELIM + id).getBytes(), (AccumuloGraph.IDDELIM + label).getBytes());
vertexWriter.addMutation(m);
m = new Mutation(src);
m.put(AccumuloGraph.OUTEDGE, (dest + AccumuloGraph.IDDELIM + id).getBytes(), (AccumuloGraph.IDDELIM + label).getBytes());
vertexWriter.addMutation(m);
Mutators.apply(edgeWriter, new EdgeMutator.Add(id, src, dest, label));
Mutators.apply(vertexWriter, new EdgeEndpointsMutator.Add(id, src, dest, label));
return new PropertyBuilder(edgeWriter, id);
}
@@ -207,22 +205,27 @@ public final class AccumuloBulkIngester {
* @throws MutationsRejectedException
*/
private void addProperty(BatchWriter writer, String id, String key, Object value) throws MutationsRejectedException {
byte[] newByteVal = AccumuloByteSerializer.serialize(value);
Mutation m = new Mutation(id);
m.put(key.getBytes(), AccumuloGraph.EMPTY, newByteVal);
writer.addMutation(m);
Mutators.apply(writer, new WritePropertyMutator(id, key, value));
}
/**
* Shutdown the bulk ingester. This flushes any outstanding writes to Accumulo and performs any remaining clean up to finalize the graph.
* Shutdown the bulk ingester. This flushes any outstanding writes to Accumulo
* and performs any remaining clean up to finalize the graph.
*
* @param compact
* a flag if this shutdown should kick off a compaction on the graph-related tables (true) or not (false) before quiting.
* a flag if this shutdown should start a compaction on the graph-related tables
* (true) or not (false) before quitting.
* @throws AccumuloException
* @throws TableNotFoundException
* @throws AccumuloSecurityException
*/
public void shutdown(boolean compact) throws AccumuloSecurityException, TableNotFoundException, AccumuloException {
public void shutdown(boolean compact) throws AccumuloSecurityException,
TableNotFoundException, AccumuloException {
// Make sure this wasn't closed already.
if (mtbw == null) {
throw new RuntimeException("Ingester was already closed");
}
mtbw.close();
mtbw = null;
@@ -266,14 +269,14 @@ public final class AccumuloBulkIngester {
* builder.finish();
* </PRE>
*/
public final class PropertyBuilder {
public static final class PropertyBuilder {
Mutation mutation;
BatchWriter writer;
final String id;
final BatchWriter writer;
PropertyBuilder(BatchWriter writer, String id) {
this.writer = writer;
this.mutation = new Mutation(id);
this.id = id;
}
/**
@@ -285,7 +288,13 @@ public final class AccumuloBulkIngester {
* @return
*/
public PropertyBuilder add(String key, Object value) {
mutation.put(key.getBytes(), AccumuloGraph.EMPTY, AccumuloByteSerializer.serialize(value));
for (Mutation m : new WritePropertyMutator(id, key, value).create()) {
try {
writer.addMutation(m);
} catch (MutationsRejectedException e) {
throw new AccumuloGraphException(e);
}
}
return this;
}
@@ -295,9 +304,7 @@ public final class AccumuloBulkIngester {
* @throws MutationsRejectedException
*/
public void finish() throws MutationsRejectedException {
if (mutation.size() > 0) {
writer.addMutation(mutation);
}
// No-op since Mutations are now added on the fly.
}
/**
@@ -306,7 +313,7 @@ public final class AccumuloBulkIngester {
* @return
*/
public String getId() {
return new String(mutation.getRow());
return id;
}
}
}

View File

@@ -26,21 +26,21 @@ import javax.xml.namespace.QName;
public final class AccumuloByteSerializer {
static final int NULL = 'n';
public static final int NULL = 'n';
static final int BYTE = 'b';
static final int SHORT = 's';
static final int CHARACTER = 'c';
static final int INTEGER = 'i';
static final int LONG = 'l';
static final int FLOAT = 'f';
static final int DOUBLE = 'd';
static final int BOOLEAN = 'o';
static final int DATE = 't';
static final int ENUM = 'e';
static final int STRING = 'a';
static final int SERIALIZABLE = 'x';
static final int QNAME = 'q';
public static final int BYTE = 'b';
public static final int SHORT = 's';
public static final int CHARACTER = 'c';
public static final int INTEGER = 'i';
public static final int LONG = 'l';
public static final int FLOAT = 'f';
public static final int DOUBLE = 'd';
public static final int BOOLEAN = 'o';
public static final int DATE = 't';
public static final int ENUM = 'e';
public static final int STRING = 'a';
public static final int SERIALIZABLE = 'x';
public static final int QNAME = 'q';
private AccumuloByteSerializer() {
@@ -53,7 +53,8 @@ public final class AccumuloByteSerializer {
}
};
public static <T> T desserialize(byte[] target) {
@SuppressWarnings("unchecked")
public static <T> T deserialize(byte[] target) {
if (target[0] == NULL) {
return null;
}
@@ -92,6 +93,7 @@ public final class AccumuloByteSerializer {
case ENUM:
try {
String[] s = new String(target, 1, target.length - 1).split(":");
@SuppressWarnings("rawtypes")
Class<? extends Enum> clz = (Class<? extends Enum>) Class.forName(s[0]);
return (T) Enum.valueOf(clz, s[1]);
} catch (ClassNotFoundException cnfe) {
@@ -107,6 +109,8 @@ public final class AccumuloByteSerializer {
} catch (ClassNotFoundException cnfe) {
throw new RuntimeException("Unexpected error deserializing object.", cnfe);
}
case NULL:
return null;
default:
throw new RuntimeException("Unexpected data type: " + (char) target[0]);
}

View File

@@ -14,71 +14,55 @@
*/
package edu.jhuapl.tinkerpop;
import java.util.Map;
import java.util.Map.Entry;
import org.apache.log4j.Logger;
import com.tinkerpop.blueprints.Direction;
import com.tinkerpop.blueprints.Edge;
import com.tinkerpop.blueprints.Vertex;
import com.tinkerpop.blueprints.util.ExceptionFactory;
import com.tinkerpop.blueprints.util.StringFactory;
/**
* TODO
*/
public class AccumuloEdge extends AccumuloElement implements Edge {
String label;
String inId;
String outId;
Vertex inVertex;
Vertex outVertex;
private static final Logger log = Logger.getLogger(AccumuloEdge.class);
AccumuloEdge(AccumuloGraph parent, String id) {
this(parent, id, null);
private String label;
private Vertex inVertex;
private Vertex outVertex;
public AccumuloEdge(GlobalInstances globals, String id) {
this(globals, id, null, null, null);
}
AccumuloEdge(AccumuloGraph parent, String id, String label) {
this(parent, id, label, (Vertex) null, (Vertex) null);
}
AccumuloEdge(AccumuloGraph parent, String id, String label, Vertex inVertex, Vertex outVertex) {
super(parent, id, Edge.class);
public AccumuloEdge(GlobalInstances globals, String id,
Vertex inVertex, Vertex outVertex, String label) {
super(globals, id, Edge.class);
this.label = label;
this.inVertex = inVertex;
this.outVertex = outVertex;
}
AccumuloEdge(AccumuloGraph parent, String id, String label, String inVertex, String outVertex) {
super(parent, id, Edge.class);
this.label = label;
this.inId = inVertex;
this.outId = outVertex;
}
@Override
public Vertex getVertex(Direction direction) throws IllegalArgumentException {
switch (direction) {
case IN:
if (inVertex == null) {
if (inId == null) {
inVertex = parent.getEdgeVertex(id, direction);
inId = inVertex.getId().toString();
} else {
inVertex = parent.getVertex(inId);
}
}
return inVertex;
case OUT:
if (outVertex == null) {
if (outId == null) {
outVertex = parent.getEdgeVertex(id, direction);
outId = outVertex.getId().toString();
} else {
outVertex = parent.getVertex(outId);
}
}
return outVertex;
case BOTH:
throw ExceptionFactory.bothIsNotSupported();
default:
throw new RuntimeException("Unexpected direction: " + direction);
if (!Direction.IN.equals(direction) && !Direction.OUT.equals(direction)) {
throw new IllegalArgumentException("Invalid direction: "+direction);
}
// The vertex information needs to be loaded.
if (inVertex == null || outVertex == null || label == null) {
log.debug("Loading information for edge: "+this);
globals.getEdgeWrapper().loadEndpointsAndLabel(this);
}
return Direction.IN.equals(direction) ? inVertex : outVertex;
}
@Override
public String getLabel() {
// TODO less special treatment for "LABEL" property...
if (label != null) {
@@ -87,31 +71,47 @@ public class AccumuloEdge extends AccumuloElement implements Edge {
return getProperty(StringFactory.LABEL);
}
@Override
public void remove() {
parent.removeEdge(this);
// Remove from named indexes.
super.removeElementFromNamedIndexes();
// If edge was removed already, forget it.
// This may happen due to self-loops...
if (!globals.getEdgeWrapper().elementExists(id)) {
return;
}
// Remove properties from key/value indexes.
Map<String, Object> props = globals.getEdgeWrapper()
.readAllProperties(this);
for (Entry<String,Object> ents : props.entrySet()) {
globals.getEdgeKeyIndexWrapper().removePropertyFromIndex(this,
ents.getKey(), ents.getValue());
}
// Get rid of the endpoints and edge themselves.
globals.getVertexWrapper().deleteEdgeEndpoints(this);
globals.getEdgeWrapper().deleteEdge(this);
// Remove element from cache.
globals.getCaches().remove(id, Edge.class);
globals.checkedFlush();
}
public String getInId() {
return inId;
public void setVertices(AccumuloVertex inVertex, AccumuloVertex outVertex) {
this.inVertex = inVertex;
this.outVertex = outVertex;
}
public String getOutId() {
return outId;
}
protected void setInId(String id){
inId = id;
}
protected void setOutId(String id){
outId = id;
}
protected void setLabel(String label){
public void setLabel(String label) {
this.label = label;
}
@Override
public String toString() {
return "[" + getId() + ":" + getVertex(Direction.OUT) + " -> " + getLabel() + " -> " + getVertex(Direction.IN) + "]";
return "[" + getId() + ":" + inVertex + " -> " + label + " -> " + outVertex + "]";
}
}

View File

@@ -14,84 +14,163 @@
*/
package edu.jhuapl.tinkerpop;
import java.util.HashMap;
import java.util.Map;
import java.util.Set;
import org.apache.accumulo.core.util.Pair;
import com.tinkerpop.blueprints.Element;
import com.tinkerpop.blueprints.Index;
import com.tinkerpop.blueprints.util.StringFactory;
import edu.jhuapl.tinkerpop.cache.PropertyCache;
/**
* TODO
*/
public abstract class AccumuloElement implements Element {
protected AccumuloGraph parent;
protected GlobalInstances globals;
protected String id;
private Class type;
private Class<? extends Element> type;
private Map<String,Pair<Long,Object>> propertiesCache;
private PropertyCache propertyCache;
protected AccumuloElement(AccumuloGraph parent, String id, Class<? extends Element> type) {
this.parent = parent;
protected AccumuloElement(GlobalInstances globals,
String id, Class<? extends Element> type) {
this.globals = globals;
this.id = id;
this.type = type;
}
public <T> T getProperty(String key) {
if (propertiesCache == null) {
// lazily create the properties cache...
/**
* Create properties cache if it doesn't exist,
* and preload any properties.
*/
private void makeCache() {
if (propertyCache == null) {
propertyCache = new PropertyCache(globals.getConfig());
// we will create it here just in case the parent does not actually
// pre-load any data. Note it also may be created in the
// cacheProperty method, as well, in the event a class pre-loads
// data before a call is made to obtain it.
propertiesCache = new HashMap<String,Pair<Long,Object>>();
parent.preloadProperties(this, type);
}
Pair<Long,Object> val = propertiesCache.get(key);
if (val != null) {
if (val.getFirst() < System.currentTimeMillis()) {
// this cached value has timed out..
propertiesCache.remove(key);
} else {
// the cached value is still good...
return (T) val.getSecond();
// Preload any keys, if needed.
String[] preloadKeys = globals.getConfig().getPreloadedProperties();
if (preloadKeys != null) {
propertyCache.putAll(globals.getElementWrapper(type)
.readProperties(this, preloadKeys));
}
}
Pair<Integer,T> pair = parent.getProperty(type, id, key);
if (pair.getFirst() != null) {
cacheProperty(key, pair.getSecond(), pair.getFirst());
}
return pair.getSecond();
}
@Override
public <T> T getProperty(String key) {
makeCache();
// Get from property cache.
T value = propertyCache.get(key);
// If not cached, get it from the backing table.
if (value == null) {
value = globals.getElementWrapper(type).readProperty(this, key);
}
// Cache the new value.
if (value != null) {
propertyCache.put(key, value);
}
return value;
}
@Override
public Set<String> getPropertyKeys() {
return parent.getPropertyKeys(type, id);
return globals.getElementWrapper(type).readPropertyKeys(this);
}
@Override
public void setProperty(String key, Object value) {
Integer timeout = parent.setProperty(type, id, key, value);
cacheProperty(key, value, timeout);
makeCache();
globals.getKeyIndexTableWrapper(type).setPropertyForIndex(this, key, value);
// MDL 31 Dec 2014: The above calls getProperty, so this
// order is important (for now).
globals.getElementWrapper(type).writeProperty(this, key, value);
globals.checkedFlush();
setPropertyInMemory(key, value);
}
/**
* Set a property but only in the instantiated object,
* not in the backing store.
* @param key
* @param value
*/
public void setPropertyInMemory(String key, Object value) {
makeCache();
propertyCache.put(key, value);
}
@Override
public <T> T removeProperty(String key) {
if (propertiesCache != null) {
// we have the cached value but we still need to pass this on to the
// parent so it can actually remove the data from the backing store.
// Since we have to do that anyway, we will use the parent's value
// instead of the cache value to to be as up-to-date as possible.
// Of course we still need to clear out the cached value...
propertiesCache.remove(key);
if (StringFactory.LABEL.equals(key) ||
Constants.LABEL.equals(key)) {
throw new AccumuloGraphException("Cannot remove the " + StringFactory.LABEL + " property.");
}
return parent.removeProperty(type, id, key);
makeCache();
T value = getProperty(key);
if (value != null) {
globals.getElementWrapper(type).clearProperty(this, key);
globals.checkedFlush();
}
globals.getKeyIndexTableWrapper(type).removePropertyFromIndex(this, key, value);
// MDL 31 Dec 2014: AccumuloGraph.removeProperty
// calls getProperty which populates the cache.
// So the order here is important (for now).
removePropertyInMemory(key);
return value;
}
/**
* Remove element from all named indexes.
* @param element
*/
protected void removeElementFromNamedIndexes() {
for (Index<? extends Element> index : globals.getIndexMetadataWrapper().getIndices()) {
((AccumuloIndex<? extends Element>) index).getWrapper().removeElementFromIndex(this);
}
}
/**
* Remove a property but only in the instantiated
* object, not the backing store.
* @param key
*/
public void removePropertyInMemory(String key) {
makeCache();
propertyCache.remove(key);
}
/**
* Return the properties currently cached in memory.
* @return
*/
public Iterable<String> getPropertyKeysInMemory() {
makeCache();
return propertyCache.keySet();
}
/**
* Retrieve a property from memory.
* @param key
* @return
*/
public Object getPropertyInMemory(String key) {
makeCache();
return propertyCache.get(key);
}
@Override
public Object getId() {
return id;
}
@Override
public boolean equals(Object obj) {
if (obj == null) {
return false;
@@ -100,25 +179,20 @@ public abstract class AccumuloElement implements Element {
} else if (!obj.getClass().equals(getClass())) {
return false;
} else {
return this.id.equals(((AccumuloElement) obj).id);
return id.equals(((AccumuloElement) obj).id);
}
}
@Override
public int hashCode() {
return getClass().hashCode() ^ id.hashCode();
}
void cacheProperty(String key, Object value, Integer timeoutMillis) {
if (timeoutMillis == null) {
// user does not want to cache data...
return;
}
if (propertiesCache == null) {
propertiesCache = new HashMap<String,Pair<Long,Object>>();
}
Pair<Long,Object> tsVal = new Pair<Long,Object>(System.currentTimeMillis() + timeoutMillis, value);
propertiesCache.put(key, tsVal);
/**
* Internal method for unit tests.
* @return
*/
PropertyCache getPropertyCache() {
return propertyCache;
}
}

View File

@@ -0,0 +1,62 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop;
import com.tinkerpop.blueprints.Features;
/**
* {@link Features} creator.
*/
public class AccumuloFeatures {
public static Features get() {
Features f = new Features();
// For simplicity, I accept all property types. They are handled in not the
// best way. To be fixed later.
f.ignoresSuppliedIds = true;
f.isPersistent = true;
f.isWrapper = false;
f.supportsBooleanProperty = true;
f.supportsDoubleProperty = true;
f.supportsDuplicateEdges = true;
f.supportsEdgeIndex = true;
f.supportsEdgeIteration = true;
f.supportsEdgeRetrieval = true;
f.supportsEdgeKeyIndex = true;
f.supportsEdgeProperties = true;
f.supportsFloatProperty = true;
f.supportsIndices = true;
f.supportsIntegerProperty = true;
f.supportsKeyIndices = true;
f.supportsLongProperty = true;
f.supportsMapProperty = true;
f.supportsMixedListProperty = true;
f.supportsPrimitiveArrayProperty = true;
f.supportsSelfLoops = true;
f.supportsSerializableObjectProperty = true;
f.supportsStringProperty = true;
f.supportsThreadedTransactions = false;
f.supportsTransactions = false;
f.supportsUniformListProperty = true;
f.supportsVertexIndex = true;
f.supportsVertexIteration = true;
f.supportsVertexKeyIndex = true;
f.supportsVertexProperties = true;
f.supportsThreadIsolatedTransactions = false;
return f;
}
}

File diff suppressed because it is too large Load Diff

View File

@@ -14,36 +14,16 @@
*/
package edu.jhuapl.tinkerpop;
import java.io.IOException;
import java.util.SortedSet;
import java.util.UUID;
import org.apache.accumulo.core.client.AccumuloException;
import org.apache.accumulo.core.client.AccumuloSecurityException;
import org.apache.accumulo.core.client.TableExistsException;
import org.apache.accumulo.core.client.TableNotFoundException;
import org.apache.accumulo.core.client.admin.TableOperations;
import org.apache.accumulo.core.data.Value;
import org.apache.hadoop.io.Text;
import com.tinkerpop.blueprints.util.ExceptionFactory;
import com.tinkerpop.blueprints.util.StringFactory;
final class AccumuloGraphUtils {
public static final Value EMPTY_VALUE = new Value(new byte[0]);
public static final Text EMPTY_TEXT = new Text("");
public static final Text ID = new Text(StringFactory.ID);
public static final Text IN = new Text("I");
public static final Text OUT = new Text("O");
public static final String toId(Object obj) {
return obj.toString();
}
public static Value toValue(String val) {
return new Value(val.getBytes());
}
public final class AccumuloGraphUtils {
/**
* Create and/or clear existing graph tables for the given configuration.
@@ -65,17 +45,17 @@ final class AccumuloGraphUtils {
// Check edge cases.
// No tables exist, and we are not allowed to create.
if (!existedBeforeClear && !cfg.isCreate()) {
if (!existedBeforeClear && !cfg.getCreate()) {
throw new IllegalArgumentException("Graph does not exist, and create option is disabled");
}
// Tables exist, and we are not clearing them.
else if (existedBeforeClear && !cfg.isClear()) {
else if (existedBeforeClear && !cfg.getClear()) {
// Do nothing.
return;
}
// We want to clear tables, so do it.
if (cfg.isClear()) {
if (cfg.getClear()) {
for (String table : cfg.getTableNames()) {
if (tableOps.exists(table)) {
tableOps.delete(table);
@@ -84,7 +64,7 @@ final class AccumuloGraphUtils {
}
// Tables existed, or we want to create them. So do it.
if (existedBeforeClear || cfg.isCreate()) {
if (existedBeforeClear || cfg.getCreate()) {
for (String table : cfg.getTableNames()) {
if (!tableOps.exists(table)) {
tableOps.create(table);
@@ -96,18 +76,49 @@ final class AccumuloGraphUtils {
}
}
} catch (AccumuloException e) {
throw new IllegalArgumentException(e);
} catch (AccumuloSecurityException e) {
throw new IllegalArgumentException(e);
} catch (IOException e) {
throw new IllegalArgumentException(e);
} catch (InterruptedException e) {
throw new IllegalArgumentException(e);
} catch (TableNotFoundException e) {
throw new IllegalArgumentException(e);
} catch (TableExistsException e) {
} catch (Exception e) {
throw new IllegalArgumentException(e);
}
}
/**
* Generate an element id.
* @return
*/
public static String generateId() {
return UUID.randomUUID().toString();
}
/**
* Ensure that the given key/value don't conflict with
* Blueprints reserved words.
* @param key
* @param value
*/
public static void validateProperty(String key, Object value) {
nullCheckProperty(key, value);
if (key.equals(StringFactory.ID)) {
throw ExceptionFactory.propertyKeyIdIsReserved();
} else if (key.equals(StringFactory.LABEL)) {
throw ExceptionFactory.propertyKeyLabelIsReservedForEdges();
} else if (value == null) {
throw ExceptionFactory.propertyValueCanNotBeNull();
}
}
/**
* Disallow null keys/values and throw appropriate
* Blueprints exceptions.
* @param key
* @param value
*/
public static void nullCheckProperty(String key, Object value) {
if (key == null) {
throw ExceptionFactory.propertyKeyCanNotBeNull();
} else if (value == null) {
throw ExceptionFactory.propertyValueCanNotBeNull();
} else if (key.trim().equals(StringFactory.EMPTY_STRING)) {
throw ExceptionFactory.propertyKeyCanNotBeEmpty();
}
}
}

View File

@@ -14,87 +14,77 @@
*/
package edu.jhuapl.tinkerpop;
import java.io.IOException;
import java.util.Iterator;
import java.util.Map.Entry;
import org.apache.accumulo.core.client.AccumuloException;
import org.apache.accumulo.core.client.AccumuloSecurityException;
import org.apache.accumulo.core.client.BatchWriter;
import org.apache.accumulo.core.client.MutationsRejectedException;
import org.apache.accumulo.core.client.Scanner;
import org.apache.accumulo.core.client.ScannerBase;
import org.apache.accumulo.core.client.TableExistsException;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Mutation;
import org.apache.accumulo.core.data.Range;
import org.apache.accumulo.core.data.Value;
import org.apache.accumulo.core.util.PeekingIterator;
import org.apache.hadoop.io.Text;
import com.tinkerpop.blueprints.CloseableIterable;
import com.tinkerpop.blueprints.Edge;
import com.tinkerpop.blueprints.Element;
import com.tinkerpop.blueprints.Index;
import com.tinkerpop.blueprints.IndexableGraph;
import edu.jhuapl.tinkerpop.tables.index.NamedIndexTableWrapper;
/**
* Accumulo-based implementation for {@link IndexableGraph}.
* @param <T>
*/
public class AccumuloIndex<T extends Element> implements Index<T> {
Class indexedType;
AccumuloGraph parent;
String indexName;
String tableName;
private final GlobalInstances globals;
private final Class<T> indexedType;
private final String indexName;
private final NamedIndexTableWrapper indexWrapper;
public AccumuloIndex(Class t, AccumuloGraph parent, String indexName) {
indexedType = t;
this.parent = parent;
public AccumuloIndex(GlobalInstances globals, String indexName, Class<T> indexedType) {
this.globals = globals;
this.indexName = indexName;
tableName = parent.config.getName() + "_index_" + indexName;// + "_" +
// t;
this.indexedType = indexedType;
try {
if (!parent.config.getConnector().tableOperations().exists(tableName)) {
parent.config.getConnector().tableOperations().create(tableName);
if (!globals.getConfig().getConnector()
.tableOperations().exists(getTableName())) {
globals.getConfig().getConnector()
.tableOperations().create(getTableName());
}
} catch (Exception e) {
throw new RuntimeException(e);
throw new AccumuloGraphException(e);
}
indexWrapper = new NamedIndexTableWrapper(globals, indexedType, indexName);
}
@Override
public String getIndexName() {
return indexName;
}
public String getTableName() {
return globals.getConfig().getNamedIndexTableName(indexName);
}
public NamedIndexTableWrapper getWrapper() {
return indexWrapper;
}
@Override
public Class<T> getIndexClass() {
return indexedType;
}
@Override
public void put(String key, Object value, Element element) {
element.setProperty(key, value);
Mutation m = new Mutation(AccumuloByteSerializer.serialize(value));
m.put(key.getBytes(), element.getId().toString().getBytes(), "".getBytes());
BatchWriter w = getWriter();
try {
w.addMutation(m);
w.flush();
} catch (MutationsRejectedException e) {
e.printStackTrace();
}
indexWrapper.setPropertyForIndex(element, key, value, true);
}
@Override
public CloseableIterable<T> get(String key, Object value) {
Scanner scan = getScanner();
byte[] id = AccumuloByteSerializer.serialize(value);
scan.setRange(new Range(new Text(id), new Text(id)));
scan.fetchColumnFamily(new Text(key));
return new IndexIterable(parent, scan, indexedType);
return indexWrapper.readElementsFromIndex(key, value);
}
@Override
public CloseableIterable<T> query(String key, Object query) {
throw new UnsupportedOperationException();
}
@Override
public long count(String key, Object value) {
CloseableIterable<T> iterable = get(key, value);
Iterator<T> iter = iterable.iterator();
@@ -107,80 +97,8 @@ public class AccumuloIndex<T extends Element> implements Index<T> {
return count;
}
public void remove(String key, Object value, Element element) {
Mutation m = new Mutation(AccumuloByteSerializer.serialize(value));
m.putDelete(key.getBytes(), element.getId().toString().getBytes());
BatchWriter w = getWriter();
try {
w.addMutation(m);
w.flush();
} catch (MutationsRejectedException e) {
e.printStackTrace();
}
}
private BatchWriter getWriter() {
return parent.getWriter(tableName);
}
private Scanner getScanner() {
return parent.getScanner(tableName);
}
public class IndexIterable implements CloseableIterable<T> {
AccumuloGraph parent;
ScannerBase scan;
boolean isClosed;
Class indexedType;
IndexIterable(AccumuloGraph parent, ScannerBase scan, Class t) {
this.scan = scan;
this.parent = parent;
isClosed = false;
indexedType = t;
}
public Iterator<T> iterator() {
if (!isClosed) {
if(indexedType.equals(Edge.class)){
return new ScannerIterable<T>(parent, scan) {
@Override
public T next(PeekingIterator<Entry<Key,Value>> iterator) {
// TODO better use of information readily
// available...
return (T) new AccumuloEdge(parent, iterator.next().getKey().getColumnQualifier().toString());
}
}.iterator();
}else{
return new ScannerIterable<T>(parent, scan) {
@Override
public T next(PeekingIterator<Entry<Key,Value>> iterator) {
// TODO better use of information readily
// available...
return (T) new AccumuloVertex(parent, iterator.next().getKey().getColumnQualifier().toString());
}
}.iterator();
}
}
return null;
}
public void close() {
if (!isClosed) {
scan.close();
isClosed = true;
}
}
}
@Override
public Class<T> getIndexClass() {
return indexedType;
public void remove(String key, Object value, Element element) {
indexWrapper.removePropertyFromIndex(element, key, value);
}
}

View File

@@ -14,38 +14,114 @@
*/
package edu.jhuapl.tinkerpop;
import java.util.Map;
import java.util.Map.Entry;
import com.tinkerpop.blueprints.CloseableIterable;
import com.tinkerpop.blueprints.Direction;
import com.tinkerpop.blueprints.Edge;
import com.tinkerpop.blueprints.Vertex;
import com.tinkerpop.blueprints.VertexQuery;
import com.tinkerpop.blueprints.util.DefaultVertexQuery;
import com.tinkerpop.blueprints.util.ExceptionFactory;
/**
* TODO
*/
public class AccumuloVertex extends AccumuloElement implements Vertex {
AccumuloVertex(AccumuloGraph parent, String id) {
super(parent, id, Vertex.class);
public AccumuloVertex(GlobalInstances globals, String id) {
super(globals, id, Vertex.class);
}
@Override
public Iterable<Edge> getEdges(Direction direction, String... labels) {
return parent.getEdges(id, direction, labels);
return globals.getVertexWrapper().getEdges(this, direction, labels);
}
@Override
public Iterable<Vertex> getVertices(Direction direction, String... labels) {
return parent.getVertices(id, direction, labels);
return globals.getVertexWrapper().getVertices(this, direction, labels);
}
@Override
public VertexQuery query() {
return new DefaultVertexQuery(this);
}
@Override
public Edge addEdge(String label, Vertex inVertex) {
return parent.addEdge(null, this, inVertex, label);
return addEdge(null, label, inVertex);
}
/**
* Add an edge as with {@link #addEdge(String, Vertex)},
* but with a specified edge id.
* @param id
* @param label
* @param inVertex
* @return
*/
public Edge addEdge(Object id, String label, Vertex inVertex) {
if (label == null) {
throw ExceptionFactory.edgeLabelCanNotBeNull();
}
if (id == null) {
id = AccumuloGraphUtils.generateId();
}
String myID = id.toString();
AccumuloEdge edge = new AccumuloEdge(globals, myID, inVertex, this, label);
// TODO we arent suppose to make sure the given edge ID doesn't already
// exist?
globals.getEdgeWrapper().writeEdge(edge);
globals.getVertexWrapper().writeEdgeEndpoints(edge);
globals.checkedFlush();
globals.getCaches().cache(edge, Edge.class);
return edge;
}
@Override
public void remove() {
parent.removeVertex(this);
globals.getCaches().remove(getId(), Vertex.class);
super.removeElementFromNamedIndexes();
// Throw exception if the element does not exist.
if (!globals.getVertexWrapper().elementExists(id)) {
throw ExceptionFactory.vertexWithIdDoesNotExist(getId());
}
// Remove properties from key/value indexes.
Map<String, Object> props = globals.getVertexWrapper()
.readAllProperties(this);
for (Entry<String,Object> ent : props.entrySet()) {
globals.getVertexKeyIndexWrapper().removePropertyFromIndex(this,
ent.getKey(), ent.getValue());
}
// Remove edges incident to this vertex.
CloseableIterable<Edge> iter = (CloseableIterable<Edge>)getEdges(Direction.BOTH);
for (Edge edge : iter) {
edge.remove();
}
iter.close();
globals.checkedFlush();
// Get rid of the vertex.
globals.getVertexWrapper().deleteVertex(this);
globals.checkedFlush();
}
@Override
public String toString() {
return "[" + getId() + "]";
}

View File

@@ -0,0 +1,46 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop;
/**
* Collect up various constants here.
* @author Michael Lieberman
*
*/
public class Constants {
private Constants() { }
/**
* Separate element ids in Accumulo entries.
*/
public static final String ID_DELIM = "__DELIM__";
public static final byte[] EMPTY = new byte[0];
/**
* Prefixes for various Accumulo entries.
*/
public static final String LABEL = "__LABEL__";
public static final String IN_EDGE = "__IN_EDGE__";
public static final String OUT_EDGE = "__OUT_EDGE__";
public static final String EXISTS = "__EXISTS__";
/**
* Type of metadata to distinguish between
* entries in the metadata table.
*/
public static enum IndexMetadataEntryType {__INDEX_KEY__, __INDEX_NAME__};
}

View File

@@ -0,0 +1,114 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop;
import org.apache.accumulo.core.client.MultiTableBatchWriter;
import org.apache.accumulo.core.client.MutationsRejectedException;
import com.tinkerpop.blueprints.Edge;
import com.tinkerpop.blueprints.Element;
import com.tinkerpop.blueprints.Vertex;
import edu.jhuapl.tinkerpop.cache.ElementCaches;
import edu.jhuapl.tinkerpop.tables.core.EdgeTableWrapper;
import edu.jhuapl.tinkerpop.tables.core.ElementTableWrapper;
import edu.jhuapl.tinkerpop.tables.core.VertexTableWrapper;
import edu.jhuapl.tinkerpop.tables.index.BaseKeyIndexTableWrapper;
import edu.jhuapl.tinkerpop.tables.index.EdgeKeyIndexTableWrapper;
import edu.jhuapl.tinkerpop.tables.index.IndexMetadataTableWrapper;
import edu.jhuapl.tinkerpop.tables.index.VertexKeyIndexTableWrapper;
/**
* Internal class gathering together instances of
* objects needed for various AccumuloGraph components.
*/
public class GlobalInstances {
private final AccumuloGraphConfiguration config;
private final MultiTableBatchWriter mtbw;
private final ElementCaches caches;
public GlobalInstances(AccumuloGraphConfiguration config,
MultiTableBatchWriter mtbw, ElementCaches caches) {
this.config = config;
this.mtbw = mtbw;
this.caches = caches;
}
public AccumuloGraphConfiguration getConfig() {
return config;
}
public MultiTableBatchWriter getMtbw() {
return mtbw;
}
public VertexTableWrapper getVertexWrapper() {
return new VertexTableWrapper(this);
}
public EdgeTableWrapper getEdgeWrapper() {
return new EdgeTableWrapper(this);
}
public VertexKeyIndexTableWrapper getVertexKeyIndexWrapper() {
return new VertexKeyIndexTableWrapper(this);
}
public EdgeKeyIndexTableWrapper getEdgeKeyIndexWrapper() {
return new EdgeKeyIndexTableWrapper(this);
}
public IndexMetadataTableWrapper getIndexMetadataWrapper() {
return new IndexMetadataTableWrapper(this);
}
public <T extends Element> ElementTableWrapper getElementWrapper(Class<T> clazz) {
if (Vertex.class.equals(clazz)) {
return getVertexWrapper();
} else if (Edge.class.equals(clazz)) {
return getEdgeWrapper();
} else {
throw new AccumuloGraphException("Unrecognized class: "+clazz);
}
}
public <T extends Element> BaseKeyIndexTableWrapper getKeyIndexTableWrapper(Class<T> clazz) {
if (Vertex.class.equals(clazz)) {
return getVertexKeyIndexWrapper();
} else if (Edge.class.equals(clazz)) {
return getEdgeKeyIndexWrapper();
} else {
throw new AccumuloGraphException("Unrecognized class: "+clazz);
}
}
public ElementCaches getCaches() {
return caches;
}
/**
* Flush the writer, if autoflush is enabled.
*/
public void checkedFlush() {
if (config.getAutoFlush()) {
try {
mtbw.flush();
} catch (MutationsRejectedException e) {
throw new AccumuloGraphException(e);
}
}
}
}

View File

@@ -1,122 +0,0 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop;
import java.lang.ref.ReferenceQueue;
import java.lang.ref.SoftReference;
import java.util.LinkedHashMap;
import java.util.Map;
import com.tinkerpop.blueprints.Element;
class LruElementCache<T extends Element> extends LinkedHashMap<String,LruElementCache<T>.Entry> implements Runnable {
private static final long serialVersionUID = 1435352624360026357L;
ReferenceQueue<T> queue;
Integer maxCapacity;
int timeout;
public LruElementCache(int timeout) {
super(32, .75f, true);
this.maxCapacity = null;
init(timeout);
}
public LruElementCache(int maxCapacity, int timeout) {
super(maxCapacity + 1, 1f, true);
this.maxCapacity = maxCapacity;
init(timeout);
}
private void init(int timeout) {
this.timeout = timeout;
this.queue = new ReferenceQueue<T>();
Thread t = new Thread(this, "lru-cache-reaper-" + System.identityHashCode(this));
t.setPriority(Thread.MIN_PRIORITY);
t.setDaemon(true);
t.start();
}
@Override
protected boolean removeEldestEntry(Map.Entry<String,LruElementCache<T>.Entry> eldest) {
if (maxCapacity != null) {
return size() > maxCapacity;
} else {
// no cap on size, but see if eldest has timed out or been gc'ed
// already...
return eldest.getValue().getElement() == null;
}
}
@Override
public synchronized Entry remove(Object id) {
return super.remove(id);
}
public synchronized void cache(T element) {
Entry entry = new Entry(element, queue, timeout);
put(element.getId().toString(), entry);
}
public synchronized T retrieve(String id) {
LruElementCache<T>.Entry entry = get(id);
if (entry == null) {
// not cached...
return null;
}
T element = (T) entry.getElement();
if (element == null) {
// gc'ed or timed out; either way...
remove(id);
}
return element;
}
@Override
public void run() {
while (true) {
try {
Entry entry = (Entry) queue.remove();
synchronized (this) {
remove(entry.id);
}
} catch (InterruptedException ie) {
throw new RuntimeException(ie);
}
}
}
final class Entry extends SoftReference<T> {
String id;
long timeout;
public Entry(T element, ReferenceQueue<T> queue, int timeout) {
super(element, queue);
this.id = element.getId().toString();
this.timeout = System.currentTimeMillis() + timeout;
}
public T getElement() {
if (System.currentTimeMillis() <= timeout) {
return get();
} else {
return null;
}
}
}
}

View File

@@ -14,7 +14,6 @@
*/
package edu.jhuapl.tinkerpop;
import java.io.Closeable;
import java.util.Iterator;
import java.util.Map.Entry;
@@ -23,15 +22,17 @@ import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Value;
import org.apache.accumulo.core.util.PeekingIterator;
import com.tinkerpop.blueprints.CloseableIterable;
import com.tinkerpop.blueprints.Element;
public abstract class ScannerIterable<T extends Element> implements Iterable<T>, Closeable {
/**
* TODO
*/
public abstract class ScannerIterable<T extends Element> implements CloseableIterable<T> {
AccumuloGraph parent;
ScannerBase scanner;
private ScannerBase scanner;
ScannerIterable(AccumuloGraph parent, ScannerBase scanner) {
this.parent = parent;
public ScannerIterable(ScannerBase scanner) {
this.scanner = scanner;
}
@@ -55,10 +56,10 @@ public abstract class ScannerIterable<T extends Element> implements Iterable<T>,
close();
}
class ScannerIterator implements Iterator<T> {
PeekingIterator<Entry<Key,Value>> iterator;
private class ScannerIterator implements Iterator<T> {
private PeekingIterator<Entry<Key,Value>> iterator;
ScannerIterator(PeekingIterator<Entry<Key,Value>> iterator) {
private ScannerIterator(PeekingIterator<Entry<Key,Value>> iterator) {
this.iterator = iterator;
}
@@ -76,7 +77,5 @@ public abstract class ScannerIterable<T extends Element> implements Iterable<T>,
public void remove() {
throw new UnsupportedOperationException();
}
}
}

View File

@@ -0,0 +1,53 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.cache;
import java.util.concurrent.TimeUnit;
import com.google.common.cache.Cache;
import com.google.common.cache.CacheBuilder;
import com.tinkerpop.blueprints.Element;
/**
* Simple cache for retrieved graph elements,
* backed by Guava's cache implementation.
*/
public class ElementCache<T extends Element> {
private Cache<Object, T> cache;
public ElementCache(int size, int timeout) {
cache = CacheBuilder.newBuilder()
.maximumSize(size)
.expireAfterAccess(timeout, TimeUnit.MILLISECONDS)
.build();
}
public void cache(T element) {
cache.put(element.getId(), element);
}
public T retrieve(Object id) {
return cache.getIfPresent(id);
}
public void remove(Object id) {
cache.invalidate(id);
}
public void clear() {
cache.invalidateAll();
}
}

View File

@@ -0,0 +1,77 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.cache;
import com.tinkerpop.blueprints.Edge;
import com.tinkerpop.blueprints.Element;
import com.tinkerpop.blueprints.Vertex;
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration;
import edu.jhuapl.tinkerpop.AccumuloGraphException;
/**
* Utility class wrapping element caches.
*/
public class ElementCaches {
private ElementCache<Vertex> vertexCache;
private ElementCache<Edge> edgeCache;
public ElementCaches(AccumuloGraphConfiguration config) {
if (config.getVertexCacheEnabled()) {
vertexCache = new ElementCache<Vertex>(config.getVertexCacheSize(),
config.getVertexCacheTimeout());
}
if (config.getEdgeCacheEnabled()) {
edgeCache = new ElementCache<Edge>(config.getEdgeCacheSize(),
config.getEdgeCacheTimeout());
}
}
public <T extends Element> void cache(T element, Class<T> clazz) {
if (pick(clazz) != null) {
pick(clazz).cache(element);
}
}
public <T extends Element> T retrieve(Object id, Class<T> clazz) {
return pick(clazz) != null ? pick(clazz).retrieve(id) : null;
}
public <T extends Element> void remove(Object id, Class<T> clazz) {
if (pick(clazz) != null) {
pick(clazz).remove(id);
}
}
public <T extends Element> void clear(Class<T> clazz) {
if (pick(clazz) != null) {
pick(clazz).clear();
}
}
@SuppressWarnings("unchecked")
private <T extends Element> ElementCache<T> pick(Class<T> clazz) {
if (Vertex.class.equals(clazz)) {
return (ElementCache<T>) vertexCache;
}
else if (Edge.class.equals(clazz)) {
return (ElementCache<T>) edgeCache;
}
else {
throw new AccumuloGraphException("Unknown element class: "+clazz);
}
}
}

View File

@@ -0,0 +1,140 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.cache;
import java.util.HashMap;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Set;
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration;
/**
* Cache for storing element properties.
* This supports a global timeout for key evictions,
* as well as per-property eviction.
* If caching is disabled for the given property,
* this does nothing (stores no values).
*/
public class PropertyCache {
private final AccumuloGraphConfiguration cfg;
private final Map<String, TimedValue> values;
public PropertyCache(AccumuloGraphConfiguration cfg) {
this.cfg = cfg;
this.values = new HashMap<String, TimedValue>();
}
public boolean containsKey(String key) {
return values.containsKey(key);
}
public Set<String> keySet() {
return values.keySet();
}
public void put(String key, Object value) {
Integer timeout = getTimeout(key);
// Don't cache anything without a specified timeout.
if (timeout == null) {
return;
}
values.put(key, new TimedValue(value,
System.currentTimeMillis() + timeout));
}
public void putAll(Map<String, Object> entries) {
if(entries==null)return;
for (Entry<String,Object> ent : entries.entrySet()) {
put(ent.getKey(), ent.getValue());
}
}
@SuppressWarnings("unchecked")
public <T> T get(String key) {
long now = System.currentTimeMillis();
TimedValue val = values.get(key);
if (val != null) {
if (val.getExpiry() != null &&
val.getExpiry() <= now) {
remove(key);
return null;
}
else {
return (T) val.getValue();
}
}
return null;
}
public void remove(String key) {
values.remove(key);
}
public void clear() {
values.clear();
}
@Override
public String toString() {
return values.toString();
}
/**
* Return the timeout for the given key.
* Checks for a key-specific timeout
* first, then the default, if any.
* Return null if no timeout.
* @param key
* @return
*/
private Integer getTimeout(String key) {
int timeout = cfg.getPropertyCacheTimeout(key);
if (timeout <= 0) {
timeout = cfg.getPropertyCacheTimeout(null);
}
return timeout > 0 ? timeout : null;
}
/**
* Value with associated expiry time.
*/
private static class TimedValue {
private final Object value;
private final Long expiry;
public TimedValue(Object value, Long expiry) {
this.value = value;
this.expiry = expiry;
}
public Object getValue() {
return value;
}
public Long getExpiry() {
return expiry;
}
@Override
public String toString() {
return "[" + value + ", " + expiry + "]";
}
}
}

View File

@@ -24,8 +24,14 @@ import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
import edu.jhuapl.tinkerpop.AccumuloGraph;
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration;
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration.InstanceType;
import edu.jhuapl.tinkerpop.AccumuloGraphException;
import edu.jhuapl.tinkerpop.Constants;
public class EdgeInputFormat extends InputFormatBase<Text,Edge> {
private static final String PREFIX = EdgeInputFormat.class.getSimpleName()+".";
private static final String GRAPH_NAME = PREFIX+"graph.name";
static AccumuloGraphConfiguration conf;
@Override
@@ -50,18 +56,17 @@ public class EdgeInputFormat extends InputFormatBase<Text,Edge> {
try {
conf = new AccumuloGraphConfiguration();
conf.setZookeeperHosts(EdgeInputFormat.getInstance(attempt).getZooKeepers());
conf.setZooKeeperHosts(EdgeInputFormat.getInstance(attempt).getZooKeepers());
conf.setInstanceName(EdgeInputFormat.getInstance(attempt).getInstanceName());
conf.setUser(EdgeInputFormat.getPrincipal(attempt));
conf.setPassword(EdgeInputFormat.getToken(attempt));
conf.setGraphName(attempt.getConfiguration().get(AccumuloGraphConfiguration.GRAPH_NAME));
conf.setGraphName(attempt.getConfiguration().get(GRAPH_NAME));
if (EdgeInputFormat.getInstance(attempt) instanceof MockInstance) {
conf.setInstanceType(InstanceType.Mock);
}
parent = AccumuloGraph.open(conf.getConfiguration());
} catch (AccumuloException e) {
// TODO Auto-generated catch block
e.printStackTrace();
throw new AccumuloGraphException(e);
}
}
@@ -80,17 +85,17 @@ public class EdgeInputFormat extends InputFormatBase<Text,Edge> {
String eid = currentKey.getRow().toString();
String colf = currentKey.getColumnFamily().toString();
switch (colf) {
case AccumuloGraph.SLABEL:
case Constants.LABEL:
currentK.set(eid);
edge.prepareId(eid);
String[] ids = currentKey.getColumnQualifier().toString().split(parent.IDDELIM);
String[] ids = currentKey.getColumnQualifier().toString().split(Constants.ID_DELIM);
edge.setSourceId(ids[1]);
edge.setDestId(ids[0]);
edge.setLabel(AccumuloByteSerializer.desserialize(entry.getValue().get()).toString());
edge.setLabel(AccumuloByteSerializer.deserialize(entry.getValue().get()).toString());
break;
default:
String propertyKey = currentKey.getColumnFamily().toString();
Object propertyValue = AccumuloByteSerializer.desserialize(entry.getValue().get());
Object propertyValue = AccumuloByteSerializer.deserialize(entry.getValue().get());
edge.prepareProperty(propertyKey, propertyValue);
}
}
@@ -105,13 +110,13 @@ public class EdgeInputFormat extends InputFormatBase<Text,Edge> {
public static void setAccumuloGraphConfiguration(Job job, AccumuloGraphConfiguration cfg) throws AccumuloSecurityException {
EdgeInputFormat.setConnectorInfo(job, cfg.getUser(), new PasswordToken(cfg.getPassword()));
EdgeInputFormat.setInputTableName(job, cfg.getEdgeTable());
EdgeInputFormat.setInputTableName(job, cfg.getEdgeTableName());
if (cfg.getInstanceType().equals(InstanceType.Mock)) {
EdgeInputFormat.setMockInstance(job, cfg.getInstance());
EdgeInputFormat.setMockInstance(job, cfg.getInstanceName());
} else {
EdgeInputFormat.setZooKeeperInstance(job, cfg.getInstance(), cfg.getZooKeeperHosts());
EdgeInputFormat.setZooKeeperInstance(job, cfg.getInstanceName(), cfg.getZooKeeperHosts());
}
job.getConfiguration().set(AccumuloGraphConfiguration.GRAPH_NAME, cfg.getName());
job.getConfiguration().set(GRAPH_NAME, cfg.getGraphName());
}

View File

@@ -25,10 +25,19 @@ import com.tinkerpop.blueprints.Element;
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration;
import edu.jhuapl.tinkerpop.AccumuloGraphException;
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration.InstanceType;
public class ElementOutputFormat extends OutputFormat<NullWritable,Element> {
private static final String PREFIX = ElementOutputFormat.class.getSimpleName()+".";
private static final String USER = "username";
private static final String PASSWORD = PREFIX+"password";
private static final String GRAPH_NAME = PREFIX+"graphName";
private static final String INSTANCE = PREFIX+"instanceName";
private static final String INSTANCE_TYPE = PREFIX+"instanceType";
private static final String ZK_HOSTS = PREFIX+"zookeeperHosts";
@Override
public RecordWriter<NullWritable,Element> getRecordWriter(TaskAttemptContext context) throws IOException, InterruptedException {
return new ElementRecordWriter(context);
@@ -43,13 +52,13 @@ public class ElementOutputFormat extends OutputFormat<NullWritable,Element> {
acc.validate();
Configuration jobconf = job.getConfiguration();
jobconf.set(AccumuloGraphConfiguration.USER, acc.getUser());
jobconf.set(AccumuloGraphConfiguration.PASSWORD, new String(acc.getPassword().array()));
jobconf.set(AccumuloGraphConfiguration.GRAPH_NAME, acc.getName());
jobconf.set(AccumuloGraphConfiguration.INSTANCE, acc.getInstance());
jobconf.set(AccumuloGraphConfiguration.INSTANCE_TYPE, acc.getInstanceType().toString());
jobconf.set(USER, acc.getUser());
jobconf.set(PASSWORD, acc.getPassword());
jobconf.set(GRAPH_NAME, acc.getGraphName());
jobconf.set(INSTANCE, acc.getInstanceName());
jobconf.set(INSTANCE_TYPE, acc.getInstanceType().toString());
if(acc.getInstanceType().equals(InstanceType.Distributed))
jobconf.set(AccumuloGraphConfiguration.ZK_HOSTS, acc.getZooKeeperHosts());
jobconf.set(ZK_HOSTS, acc.getZooKeeperHosts());
}
/**
@@ -61,18 +70,18 @@ public class ElementOutputFormat extends OutputFormat<NullWritable,Element> {
return new NullOutputFormat<Text,Mutation>().getOutputCommitter(context);
}
class ElementRecordWriter extends RecordWriter<NullWritable,Element> {
static class ElementRecordWriter extends RecordWriter<NullWritable,Element> {
AccumuloGraphConfiguration config;
protected ElementRecordWriter(TaskAttemptContext context) {
config = new AccumuloGraphConfiguration();
Configuration jobconf = context.getConfiguration();
config.setUser(jobconf.get(AccumuloGraphConfiguration.USER));
config.setPassword(jobconf.get(AccumuloGraphConfiguration.PASSWORD));
config.setGraphName(jobconf.get(AccumuloGraphConfiguration.GRAPH_NAME));
config.setInstanceName(jobconf.get(AccumuloGraphConfiguration.INSTANCE));
config.setInstanceType(InstanceType.valueOf(jobconf.get(AccumuloGraphConfiguration.INSTANCE_TYPE)));
config.setZookeeperHosts(jobconf.get(AccumuloGraphConfiguration.ZK_HOSTS));
config.setUser(jobconf.get(USER));
config.setPassword(jobconf.get(PASSWORD));
config.setGraphName(jobconf.get(GRAPH_NAME));
config.setInstanceName(jobconf.get(INSTANCE));
config.setInstanceType(InstanceType.valueOf(jobconf.get(INSTANCE_TYPE)));
config.setZooKeeperHosts(jobconf.get(ZK_HOSTS));
}
@@ -84,9 +93,9 @@ public class ElementOutputFormat extends OutputFormat<NullWritable,Element> {
try {
if (bw == null) {
if (ele instanceof MapReduceVertex) {
bw = config.getConnector().createBatchWriter(config.getVertexTable(), config.getBatchWriterConfig());
bw = config.getConnector().createBatchWriter(config.getVertexTableName(), config.getBatchWriterConfig());
} else {
bw = config.getConnector().createBatchWriter(config.getEdgeTable(), config.getBatchWriterConfig());
bw = config.getConnector().createBatchWriter(config.getEdgeTableName(), config.getBatchWriterConfig());
}
}
@@ -97,9 +106,7 @@ public class ElementOutputFormat extends OutputFormat<NullWritable,Element> {
bw.addMutation(mut);
} catch (TableNotFoundException | AccumuloException | AccumuloSecurityException e) {
// TODO Auto-generated catch block
e.printStackTrace();
throw new RuntimeException(e);
throw new AccumuloGraphException(e);
}
}
@@ -109,7 +116,7 @@ public class ElementOutputFormat extends OutputFormat<NullWritable,Element> {
try {
bw.close();
} catch (MutationsRejectedException e) {
e.printStackTrace();
throw new AccumuloGraphException(e);
}
}
}

View File

@@ -17,7 +17,6 @@ package edu.jhuapl.tinkerpop.mapreduce;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Map;
@@ -65,6 +64,7 @@ public abstract class MapReduceElement implements Element, WritableComparable<Ma
return id;
}
@SuppressWarnings("unchecked")
@Override
public <T> T getProperty(String key) {
@@ -108,7 +108,7 @@ public abstract class MapReduceElement implements Element, WritableComparable<Ma
String key = in.readUTF();
byte[] data = new byte[in.readInt()];
in.readFully(data);
Object val = AccumuloByteSerializer.desserialize(data);
Object val = AccumuloByteSerializer.deserialize(data);
properties.put(key, val);
}
@@ -117,7 +117,7 @@ public abstract class MapReduceElement implements Element, WritableComparable<Ma
String key = in.readUTF();
byte[] data = new byte[in.readInt()];
in.readFully(data);
Object val = AccumuloByteSerializer.desserialize(data);
Object val = AccumuloByteSerializer.deserialize(data);
newProperties.put(key, val);
}

View File

@@ -1,7 +0,0 @@
package edu.jhuapl.tinkerpop.mapreduce;
import org.apache.accumulo.core.client.mapreduce.AccumuloOutputFormat;
public class NewElementOutputFormat extends AccumuloOutputFormat{
}

View File

@@ -12,26 +12,30 @@ import org.apache.accumulo.core.client.mock.MockInstance;
import org.apache.accumulo.core.client.security.tokens.PasswordToken;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Value;
import org.apache.commons.configuration.Configuration;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.InputSplit;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.RecordReader;
import org.apache.hadoop.mapreduce.TaskAttemptContext;
import com.tinkerpop.blueprints.Graph;
import com.tinkerpop.blueprints.Vertex;
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
import edu.jhuapl.tinkerpop.AccumuloGraph;
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration;
import edu.jhuapl.tinkerpop.AccumuloGraphException;
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration.InstanceType;
import edu.jhuapl.tinkerpop.Constants;
public class VertexInputFormat extends InputFormatBase<Text,Vertex> {
static AccumuloGraphConfiguration conf;
private static final String PREFIX = VertexInputFormat.class.getSimpleName()+".";
private static final String GRAPH_NAME = PREFIX+"graph.name";
@Override
public RecordReader<Text,Vertex> createRecordReader(InputSplit split, TaskAttemptContext attempt) throws IOException, InterruptedException {
public RecordReader<Text,Vertex> createRecordReader(InputSplit split,
TaskAttemptContext attempt) throws IOException, InterruptedException {
return new VertexRecordReader();
}
@@ -52,21 +56,19 @@ public class VertexInputFormat extends InputFormatBase<Text,Vertex> {
try {
conf = new AccumuloGraphConfiguration();
conf.setZookeeperHosts(VertexInputFormat.getInstance(attempt).getZooKeepers());
conf.setZooKeeperHosts(VertexInputFormat.getInstance(attempt).getZooKeepers());
conf.setInstanceName(VertexInputFormat.getInstance(attempt).getInstanceName());
conf.setUser(VertexInputFormat.getPrincipal(attempt));
conf.setPassword(VertexInputFormat.getToken(attempt));
conf.setGraphName(attempt.getConfiguration().get(AccumuloGraphConfiguration.GRAPH_NAME));
conf.setGraphName(attempt.getConfiguration().get(GRAPH_NAME));
if (VertexInputFormat.getInstance(attempt) instanceof MockInstance) {
conf.setInstanceType(InstanceType.Mock);
}
parent = AccumuloGraph.open(conf.getConfiguration());
} catch (AccumuloException e) {
// TODO Auto-generated catch block
e.printStackTrace();
throw new AccumuloGraphException(e);
}
}
@Override
@@ -83,23 +85,23 @@ public class VertexInputFormat extends InputFormatBase<Text,Vertex> {
String vid = currentKey.getRow().toString();
String colf = currentKey.getColumnFamily().toString();
switch (colf) {
case AccumuloGraph.SLABEL:
case Constants.LABEL:
currentK.set(vid);
vertex.prepareId(vid);
break;
case AccumuloGraph.SINEDGE:
String[] parts = currentKey.getColumnQualifier().toString().split(AccumuloGraph.IDDELIM);
case Constants.IN_EDGE:
String[] parts = currentKey.getColumnQualifier().toString().split(Constants.ID_DELIM);
String label = new String(entry.getValue().get());
vertex.prepareEdge(parts[1], parts[0], label, vid);
break;
case AccumuloGraph.SOUTEDGE:
parts = currentKey.getColumnQualifier().toString().split(AccumuloGraph.IDDELIM);
case Constants.OUT_EDGE:
parts = currentKey.getColumnQualifier().toString().split(Constants.ID_DELIM);
label = new String(entry.getValue().get());
vertex.prepareEdge(parts[1], vid, label, parts[0]);
break;
default:
String propertyKey = currentKey.getColumnFamily().toString();
Object propertyValue = AccumuloByteSerializer.desserialize(entry.getValue().get());
Object propertyValue = AccumuloByteSerializer.deserialize(entry.getValue().get());
vertex.prepareProperty(propertyKey, propertyValue);
}
}
@@ -114,13 +116,13 @@ public class VertexInputFormat extends InputFormatBase<Text,Vertex> {
public static void setAccumuloGraphConfiguration(Job job, AccumuloGraphConfiguration cfg) throws AccumuloSecurityException {
VertexInputFormat.setConnectorInfo(job, cfg.getUser(), new PasswordToken(cfg.getPassword()));
VertexInputFormat.setInputTableName(job, cfg.getVertexTable());
VertexInputFormat.setInputTableName(job, cfg.getVertexTableName());
if (cfg.getInstanceType().equals(InstanceType.Mock)) {
VertexInputFormat.setMockInstance(job, cfg.getInstance());
VertexInputFormat.setMockInstance(job, cfg.getInstanceName());
} else {
VertexInputFormat.setZooKeeperInstance(job, cfg.getInstance(), cfg.getZooKeeperHosts());
VertexInputFormat.setZooKeeperInstance(job, cfg.getInstanceName(), cfg.getZooKeeperHosts());
}
job.getConfiguration().set(AccumuloGraphConfiguration.GRAPH_NAME, cfg.getName());
job.getConfiguration().set(GRAPH_NAME, cfg.getGraphName());
}
}

View File

@@ -0,0 +1,22 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.mutator;
import org.apache.accumulo.core.data.Mutation;
public interface Mutator {
public Iterable<Mutation> create();
}

View File

@@ -0,0 +1,53 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.mutator;
import java.util.LinkedList;
import java.util.List;
import org.apache.accumulo.core.client.BatchDeleter;
import org.apache.accumulo.core.client.BatchWriter;
import org.apache.accumulo.core.client.MutationsRejectedException;
import org.apache.accumulo.core.data.Range;
import com.tinkerpop.blueprints.Element;
import edu.jhuapl.tinkerpop.AccumuloGraphException;
public class Mutators {
public static void apply(BatchWriter writer, Mutator mut) {
try {
writer.addMutations(mut.create());
} catch (MutationsRejectedException e) {
throw new AccumuloGraphException(e);
}
}
public static void deleteElementRanges(BatchDeleter deleter, Element... elements) {
List<Range> ranges = new LinkedList<Range>();
for (Element element : elements) {
ranges.add(new Range(element.getId().toString()));
}
deleter.setRanges(ranges);
try {
deleter.delete();
} catch (Exception e) {
throw new AccumuloGraphException(e);
}
}
}

View File

@@ -0,0 +1,41 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.mutator.edge;
import com.tinkerpop.blueprints.Direction;
import com.tinkerpop.blueprints.Edge;
import edu.jhuapl.tinkerpop.mutator.Mutator;
public abstract class BaseEdgeMutator implements Mutator {
protected final String id;
protected final String outVertexId;
protected final String inVertexId;
protected final String label;
public BaseEdgeMutator(Edge edge) {
this(edge.getId().toString(),
edge.getVertex(Direction.OUT).getId().toString(),
edge.getVertex(Direction.IN).getId().toString(),
edge.getLabel());
}
public BaseEdgeMutator(String id, String outVertexId, String inVertexId, String label) {
this.id = id;
this.outVertexId = outVertexId;
this.inVertexId = inVertexId;
this.label = label;
}
}

View File

@@ -0,0 +1,78 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.mutator.edge;
import org.apache.accumulo.core.data.Mutation;
import com.google.common.collect.Lists;
import com.tinkerpop.blueprints.Edge;
import edu.jhuapl.tinkerpop.Constants;
public class EdgeEndpointsMutator {
private EdgeEndpointsMutator() {
}
public static class Add extends BaseEdgeMutator {
public Add(Edge edge) {
super(edge);
}
public Add(String id, String outVertexId, String inVertexId, String label) {
super(id, outVertexId, inVertexId, label);
}
@Override
public Iterable<Mutation> create() {
Mutation in = new Mutation(inVertexId);
in.put(Constants.IN_EDGE.getBytes(),
(outVertexId + Constants.ID_DELIM + id).getBytes(),
(Constants.ID_DELIM + label).getBytes());
Mutation out = new Mutation(outVertexId);
out.put(Constants.OUT_EDGE.getBytes(),
(inVertexId + Constants.ID_DELIM + id).getBytes(),
(Constants.ID_DELIM + label).getBytes());
return Lists.newArrayList(in, out);
}
}
public static class Delete extends BaseEdgeMutator {
public Delete(Edge edge) {
super(edge);
}
public Delete(String id, String outVertexId, String inVertexId, String label) {
super(id, outVertexId, inVertexId, label);
}
@Override
public Iterable<Mutation> create() {
Mutation in = new Mutation(inVertexId);
in.putDelete(Constants.IN_EDGE.getBytes(),
(outVertexId + Constants.ID_DELIM + id).getBytes());
Mutation out = new Mutation(outVertexId);
out.putDelete(Constants.OUT_EDGE.getBytes(),
(inVertexId + Constants.ID_DELIM + id).getBytes());
return Lists.newArrayList(in, out);
}
}
}

View File

@@ -0,0 +1,67 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.mutator.edge;
import org.apache.accumulo.core.data.Mutation;
import com.google.common.collect.Lists;
import com.tinkerpop.blueprints.Edge;
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
import edu.jhuapl.tinkerpop.Constants;
public final class EdgeMutator {
public static class Add extends BaseEdgeMutator {
public Add(Edge edge) {
super(edge);
}
public Add(String id, String outVertexId, String inVertexId, String label) {
super(id, outVertexId, inVertexId, label);
}
@Override
public Iterable<Mutation> create() {
Mutation m = new Mutation(id);
m.put(Constants.LABEL.getBytes(),
(inVertexId + Constants.ID_DELIM + outVertexId).getBytes(),
AccumuloByteSerializer.serialize(label));
return Lists.newArrayList(m);
}
}
public static class Delete extends BaseEdgeMutator {
public Delete(Edge edge) {
super(edge);
}
public Delete(String id, String outVertexId, String inVertexId, String label) {
super(id, outVertexId, inVertexId, label);
}
@Override
public Iterable<Mutation> create() {
Mutation m = new Mutation(id);
m.putDelete(Constants.LABEL.getBytes(),
(inVertexId + Constants.ID_DELIM + outVertexId).getBytes());
return Lists.newArrayList(m);
}
}
}

View File

@@ -0,0 +1,76 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.mutator.index;
import org.apache.accumulo.core.data.Mutation;
import com.google.common.collect.Lists;
import com.tinkerpop.blueprints.Element;
import edu.jhuapl.tinkerpop.Constants;
import edu.jhuapl.tinkerpop.Constants.IndexMetadataEntryType;
import edu.jhuapl.tinkerpop.mutator.Mutator;
/**
* Mutators for index metadata table entries.
*/
public class IndexMetadataMutator {
private IndexMetadataMutator() { }
public static class Add implements Mutator {
private final String key;
private final Class<? extends Element> elementClass;
private final IndexMetadataEntryType entryType;
public Add(String key, Class<? extends Element> elementClass,
IndexMetadataEntryType entryType) {
this.key = key;
this.elementClass = elementClass;
this.entryType = entryType;
}
@Override
public Iterable<Mutation> create() {
Mutation m = new Mutation(key);
m.put(entryType.name().getBytes(),
elementClass.getName().getBytes(), Constants.EMPTY);
return Lists.newArrayList(m);
}
}
public static class Delete implements Mutator {
private final String key;
private final Class<? extends Element> elementClass;
private final IndexMetadataEntryType entryType;
public Delete(String indexName, Class<? extends Element> elementClass,
IndexMetadataEntryType entryType) {
this.key = indexName;
this.elementClass = elementClass;
this.entryType = entryType;
}
@Override
public Iterable<Mutation> create() {
Mutation m = new Mutation(key);
m.putDelete(entryType.name().getBytes(),
elementClass.getName().getBytes());
return Lists.newArrayList(m);
}
}
}

View File

@@ -0,0 +1,75 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.mutator.index;
import org.apache.accumulo.core.data.Mutation;
import com.google.common.collect.Lists;
import com.tinkerpop.blueprints.Element;
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
import edu.jhuapl.tinkerpop.Constants;
import edu.jhuapl.tinkerpop.mutator.Mutator;
/**
* Mutators for vertex/edge index tables.
*/
public class IndexValueMutator {
private IndexValueMutator() { }
public static class Add implements Mutator {
private final Element element;
private final String key;
private final Object value;
public Add(Element element, String key, Object value) {
this.element = element;
this.key = key;
this.value = value;
}
@Override
public Iterable<Mutation> create() {
byte[] bytes = AccumuloByteSerializer.serialize(value);
Mutation m = new Mutation(bytes);
m.put(key.getBytes(), element.getId().toString()
.getBytes(), Constants.EMPTY);
return Lists.newArrayList(m);
}
}
public static class Delete implements Mutator {
private final Element element;
private final String key;
private final Object value;
public Delete(Element element, String key, Object value) {
this.element = element;
this.key = key;
this.value = value;
}
@Override
public Iterable<Mutation> create() {
byte[] bytes = AccumuloByteSerializer.serialize(value);
Mutation m = new Mutation(bytes);
m.putDelete(key, element.getId().toString());
return Lists.newArrayList(m);
}
}
}

View File

@@ -0,0 +1,28 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.mutator.property;
import edu.jhuapl.tinkerpop.mutator.Mutator;
public abstract class BasePropertyMutator implements Mutator {
protected final String id;
protected final String key;
public BasePropertyMutator(String id, String key) {
this.id = id;
this.key = key;
}
}

View File

@@ -0,0 +1,34 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.mutator.property;
import org.apache.accumulo.core.data.Mutation;
import com.google.common.collect.Lists;
import edu.jhuapl.tinkerpop.Constants;
public class ClearPropertyMutator extends BasePropertyMutator {
public ClearPropertyMutator(String id, String key) {
super(id, key);
}
@Override
public Iterable<Mutation> create() {
Mutation m = new Mutation(id);
m.putDelete(key.getBytes(), Constants.EMPTY);
return Lists.newArrayList(m);
}
}

View File

@@ -0,0 +1,39 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.mutator.property;
import org.apache.accumulo.core.data.Mutation;
import com.google.common.collect.Lists;
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
import edu.jhuapl.tinkerpop.Constants;
public class WritePropertyMutator extends BasePropertyMutator {
private final Object value;
public WritePropertyMutator(String id, String key, Object value) {
super(id, key);
this.value = value;
}
@Override
public Iterable<Mutation> create() {
byte[] bytes = AccumuloByteSerializer.serialize(value);
Mutation m = new Mutation(id);
m.put(key.getBytes(), Constants.EMPTY, bytes);
return Lists.newArrayList(m);
}
}

View File

@@ -0,0 +1,38 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.mutator.vertex;
import org.apache.accumulo.core.data.Mutation;
import com.google.common.collect.Lists;
import edu.jhuapl.tinkerpop.Constants;
import edu.jhuapl.tinkerpop.mutator.Mutator;
public final class AddVertexMutator implements Mutator {
private final String id;
public AddVertexMutator(String id) {
this.id = id;
}
@Override
public Iterable<Mutation> create() {
Mutation m = new Mutation(id);
m.put(Constants.LABEL.getBytes(),
Constants.EXISTS.getBytes(), Constants.EMPTY);
return Lists.newArrayList(m);
}
}

View File

@@ -0,0 +1,33 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.parser;
import edu.jhuapl.tinkerpop.AccumuloEdge;
import edu.jhuapl.tinkerpop.GlobalInstances;
/**
* Edge-specific index parser.
*/
public class EdgeIndexParser extends ElementIndexParser<AccumuloEdge> {
public EdgeIndexParser(GlobalInstances globals) {
super(globals);
}
@Override
protected AccumuloEdge instantiate(String id) {
return new AccumuloEdge(globals, id);
}
}

View File

@@ -0,0 +1,70 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.parser;
import java.util.Map.Entry;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Value;
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
import edu.jhuapl.tinkerpop.AccumuloEdge;
import edu.jhuapl.tinkerpop.AccumuloGraphException;
import edu.jhuapl.tinkerpop.AccumuloVertex;
import edu.jhuapl.tinkerpop.Constants;
import edu.jhuapl.tinkerpop.GlobalInstances;
/**
* TODO
*/
public class EdgeParser extends ElementParser<AccumuloEdge> {
public EdgeParser(GlobalInstances globals) {
super(globals);
}
@Override
public AccumuloEdge parse(String id, Iterable<Entry<Key,Value>> entries) {
AccumuloEdge edge = makeEdge(id, entries);
setInMemoryProperties(edge, entries);
return edge;
}
/**
* Make and return an edge object. If the entries
* contain label/endpoint information, set those too.
* @param id
* @param entries
* @return
*/
private AccumuloEdge makeEdge(String id, Iterable<Entry<Key,Value>> entries) {
for (Entry<Key, Value> entry : entries) {
String cf = entry.getKey().getColumnFamily().toString();
if (Constants.LABEL.equals(cf)) {
String cq = entry.getKey().getColumnQualifier().toString();
String[] parts = cq.split(Constants.ID_DELIM);
String inVertexId = parts[0];
String outVertexId = parts[1];
String label = AccumuloByteSerializer.deserialize(entry.getValue().get());
return new AccumuloEdge(globals, id,
new AccumuloVertex(globals, inVertexId),
new AccumuloVertex(globals, outVertexId), label);
}
}
// This should not happen.
throw new AccumuloGraphException("Unable to parse edge from entries");
}
}

View File

@@ -0,0 +1,72 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.parser;
import java.util.Iterator;
import java.util.Map.Entry;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Value;
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
import edu.jhuapl.tinkerpop.AccumuloElement;
import edu.jhuapl.tinkerpop.AccumuloGraphException;
import edu.jhuapl.tinkerpop.GlobalInstances;
/**
* Parser for elements based on an index table.
*/
public abstract class ElementIndexParser<T extends AccumuloElement>
implements EntryParser<T> {
protected final GlobalInstances globals;
public ElementIndexParser(GlobalInstances globals) {
this.globals = globals;
}
@Override
public T parse(Iterable<Entry<Key, Value>> entries) {
Iterator<Entry<Key, Value>> it = entries.iterator();
if (it.hasNext()) {
Entry<Key, Value> entry = it.next();
if (it.hasNext()) {
throw new AccumuloGraphException("Unexpected multiple entries for index table");
}
String id = entry.getKey().getColumnQualifierData().toString();
T element = instantiate(id);
// While we're here, read the property key/value.
String key = entry.getKey().getColumnFamily().toString();
Object value = AccumuloByteSerializer.deserialize(entry.getKey().getRow().getBytes());
element.setPropertyInMemory(key, value);
return element;
}
else {
throw new AccumuloGraphException("No index table entries found");
}
}
/**
* Instantiate an object to be returned.
* @param id
* @return
*/
protected abstract T instantiate(String id);
}

View File

@@ -0,0 +1,65 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.parser;
import java.util.Map;
import java.util.Map.Entry;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Value;
import edu.jhuapl.tinkerpop.AccumuloElement;
import edu.jhuapl.tinkerpop.GlobalInstances;
/**
* TODO
*/
public abstract class ElementParser<T extends AccumuloElement> implements EntryParser<T> {
protected GlobalInstances globals;
public ElementParser(GlobalInstances globals) {
this.globals = globals;
}
@Override
public T parse(Iterable<Entry<Key, Value>> entries) {
String id = entries.iterator().next().getKey().getRow().toString();
return parse(id, entries);
}
/**
* Given the element id and set of entries,
* create the type of element. This can leverage
* the property loader, etc.
* @param id
* @param entries
* @return
*/
public abstract T parse(String id, Iterable<Entry<Key, Value>> entries);
/**
* Parse out the property entries and set them for the given element.
* @param element
* @param entries
*/
protected void setInMemoryProperties(T element, Iterable<Entry<Key, Value>> entries) {
Map<String, Object> props = new PropertyParser().parse(entries);
if(props == null) return;
for (Entry<String, Object> ent : props.entrySet()) {
element.setPropertyInMemory(ent.getKey(), ent.getValue());
}
}
}

View File

@@ -12,30 +12,17 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop;
package edu.jhuapl.tinkerpop.parser;
import java.util.Map.Entry;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Value;
public class DataConverter {
/**
* TODO
*/
public interface EntryParser<T> {
public static Value toValue(Object data) {
if (data instanceof String) {
String val = "S" + data;
return new Value(val.getBytes());
} else if (data instanceof Value) {
return (Value) data;
}
throw new UnsupportedOperationException("TODO: handle " + data.getClass());
}
@SuppressWarnings("unchecked")
public static <T> T fromValue(Value value) {
byte[] data = value.get();
switch (data[0]) {
case 'S':
return (T) new String(data, 1, data.length - 1);
}
throw new UnsupportedOperationException("TODO: handle type: " + data[0]);
}
public T parse(Iterable<Entry<Key, Value>> entries);
}

View File

@@ -0,0 +1,44 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.parser;
import com.tinkerpop.blueprints.Element;
import com.tinkerpop.blueprints.IndexableGraph;
import com.tinkerpop.blueprints.KeyIndexableGraph;
/**
* An indexed item. For {@link IndexableGraph},
* the key is the index name. For {@link KeyIndexableGraph}
* the key is the indexed key.
* @author Michael Lieberman
*
*/
public class IndexedItem {
private final String key;
private final Class<? extends Element> elementClass;
public IndexedItem(String key, Class<? extends Element> elementClass) {
this.key = key;
this.elementClass = elementClass;
}
public String getKey() {
return key;
}
public Class<? extends Element> getElementClass() {
return elementClass;
}
}

View File

@@ -0,0 +1,91 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.parser;
import java.util.ArrayList;
import java.util.List;
import java.util.Map.Entry;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Value;
import com.tinkerpop.blueprints.Edge;
import com.tinkerpop.blueprints.Element;
import com.tinkerpop.blueprints.IndexableGraph;
import com.tinkerpop.blueprints.KeyIndexableGraph;
import com.tinkerpop.blueprints.Vertex;
import edu.jhuapl.tinkerpop.AccumuloGraphException;
/**
* Entry parser for index metadata. The format
* is the same for both {@link IndexableGraph}
* and {@link KeyIndexableGraph} functionality.
* For the former, this parser returns names of
* indexes. For the latter, the parser returns
* indexed keys.
*/
public class IndexedItemsListParser implements EntryParser<List<IndexedItem>> {
private final Class<? extends Element> elementClass;
/**
* Constructor to return all items regardless
* of element class.
*/
public IndexedItemsListParser() {
this(Element.class);
}
/**
* Create a parser for items of the specified element class.
* This may be Vertex, Edge, or Element.
* @param elementClass
*/
public IndexedItemsListParser(Class<? extends Element> elementClass) {
// Validate element class.
if (!Vertex.class.equals(elementClass) &&
!Edge.class.equals(elementClass) &&
!Element.class.equals(elementClass)) {
throw new IllegalArgumentException("elementClass must be Vertex, Edge or Element");
}
this.elementClass = elementClass;
}
@SuppressWarnings("unchecked")
@Override
public List<IndexedItem> parse(Iterable<Entry<Key,Value>> entries) {
List<IndexedItem> items = new ArrayList<IndexedItem>();
for (Entry<Key, Value> entry : entries) {
Class<? extends Element> clazz;
try {
clazz = (Class<? extends Element>) Class.forName(entry.getKey()
.getColumnQualifier().toString());
} catch (ClassNotFoundException e) {
throw new AccumuloGraphException(e);
}
if (Element.class.equals(elementClass) ||
elementClass.equals(clazz)) {
IndexedItem item = new IndexedItem(entry.getKey()
.getRow().toString(), clazz);
items.add(item);
}
}
return items;
}
}

View File

@@ -0,0 +1,66 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.parser;
import java.util.HashMap;
import java.util.Map;
import java.util.Map.Entry;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Value;
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
import edu.jhuapl.tinkerpop.Constants;
/**
* TODO
*/
public class PropertyParser implements EntryParser<Map<String, Object>> {
@Override
public Map<String,Object> parse(Iterable<Entry<Key,Value>> entries) {
Map<String, Object> props = null;
for (Entry<Key, Value> entry : entries) {
if (props == null) {
props = new HashMap<String, Object>();
}
Key key = entry.getKey();
if (!isMetaKey(key)) {
String attr = key.getColumnFamily().toString();
Object value = AccumuloByteSerializer.deserialize(entry.getValue().get());
props.put(attr, value);
}
}
return props;
}
/**
* Test whether the given Accumulo key represents a
* metadata key (e.g. existence, edge endpoint, etc),
* rather than a property.
* @param key
* @return
*/
private static boolean isMetaKey(Key key) {
String cf = key.getColumnFamily().toString();
return Constants.LABEL.equals(cf) ||
Constants.IN_EDGE.equals(cf) ||
Constants.OUT_EDGE.equals(cf);
}
}

View File

@@ -0,0 +1,33 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.parser;
import edu.jhuapl.tinkerpop.AccumuloVertex;
import edu.jhuapl.tinkerpop.GlobalInstances;
/**
* Vertex-specific index parser.
*/
public class VertexIndexParser extends ElementIndexParser<AccumuloVertex> {
public VertexIndexParser(GlobalInstances globals) {
super(globals);
}
@Override
protected AccumuloVertex instantiate(String id) {
return new AccumuloVertex(globals, id);
}
}

View File

@@ -12,29 +12,29 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop;
package edu.jhuapl.tinkerpop.parser;
import java.util.Map.Entry;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Value;
public enum EntryLocation {
import edu.jhuapl.tinkerpop.AccumuloVertex;
import edu.jhuapl.tinkerpop.GlobalInstances;
Row, ColF, ColQ, Value;
/**
* TODO
*/
public class VertexParser extends ElementParser<AccumuloVertex> {
public String extract(Entry<Key,Value> entry) {
switch (this) {
case Row:
return entry.getKey().getRow().toString();
case ColF:
return entry.getKey().getColumnFamily().toString();
case ColQ:
return entry.getKey().getColumnQualifier().toString();
case Value:
return new String(entry.getValue().get());
default:
throw new AccumuloGraphException("Unexpected type: " + this);
}
public VertexParser(GlobalInstances globals) {
super(globals);
}
@Override
public AccumuloVertex parse(String id, Iterable<Entry<Key,Value>> entries) {
AccumuloVertex vertex = new AccumuloVertex(globals, id);
setInMemoryProperties(vertex, entries);
return vertex;
}
}

View File

@@ -0,0 +1,90 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.tables;
import java.util.Collections;
import java.util.Map.Entry;
import org.apache.accumulo.core.client.BatchDeleter;
import org.apache.accumulo.core.client.BatchScanner;
import org.apache.accumulo.core.client.BatchWriter;
import org.apache.accumulo.core.client.Scanner;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Range;
import org.apache.accumulo.core.data.Value;
import edu.jhuapl.tinkerpop.AccumuloGraphException;
import edu.jhuapl.tinkerpop.GlobalInstances;
/**
* Table wrapper with common functionality.
*/
public abstract class BaseTableWrapper {
protected GlobalInstances globals;
private String tableName;
public BaseTableWrapper(GlobalInstances globals, String tableName) {
this.globals = globals;
this.tableName = tableName;
}
protected Scanner getScanner() {
try {
return globals.getConfig().getConnector().createScanner(tableName,
globals.getConfig().getAuthorizations());
} catch (Exception e) {
throw new AccumuloGraphException(e);
}
}
protected BatchScanner getBatchScanner() {
try {
BatchScanner scanner = globals.getConfig().getConnector().createBatchScanner(tableName,
globals.getConfig().getAuthorizations(), globals.getConfig().getQueryThreads());
scanner.setRanges(Collections.singletonList(new Range()));
return scanner;
} catch (Exception e) {
throw new AccumuloGraphException(e);
}
}
protected BatchWriter getWriter() {
try {
return globals.getMtbw().getBatchWriter(tableName);
} catch (Exception e) {
throw new AccumuloGraphException(e);
}
}
protected BatchDeleter getDeleter() {
try {
return globals.getConfig().getConnector().createBatchDeleter(tableName,
globals.getConfig().getAuthorizations(), globals.getConfig().getMaxWriteThreads(),
globals.getConfig().getBatchWriterConfig());
} catch (Exception e) {
throw new AccumuloGraphException(e);
}
}
public void dump() {
System.out.println("Dump of table "+tableName+":");
Scanner s = getScanner();
for (Entry<Key, Value> entry : s) {
System.out.println(" "+entry);
}
}
}

View File

@@ -0,0 +1,177 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.tables.core;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import java.util.Map.Entry;
import java.util.regex.Pattern;
import org.apache.accumulo.core.client.BatchScanner;
import org.apache.accumulo.core.client.IteratorSetting;
import org.apache.accumulo.core.client.Scanner;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Range;
import org.apache.accumulo.core.data.Value;
import org.apache.accumulo.core.iterators.user.RegExFilter;
import org.apache.accumulo.core.util.PeekingIterator;
import org.apache.hadoop.io.Text;
import com.tinkerpop.blueprints.CloseableIterable;
import com.tinkerpop.blueprints.Edge;
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
import edu.jhuapl.tinkerpop.AccumuloEdge;
import edu.jhuapl.tinkerpop.AccumuloGraphException;
import edu.jhuapl.tinkerpop.AccumuloGraphUtils;
import edu.jhuapl.tinkerpop.AccumuloVertex;
import edu.jhuapl.tinkerpop.Constants;
import edu.jhuapl.tinkerpop.GlobalInstances;
import edu.jhuapl.tinkerpop.ScannerIterable;
import edu.jhuapl.tinkerpop.mutator.Mutators;
import edu.jhuapl.tinkerpop.mutator.edge.EdgeMutator;
import edu.jhuapl.tinkerpop.parser.EdgeParser;
/**
* Wrapper around {@link Edge} tables.
*/
public class EdgeTableWrapper extends ElementTableWrapper {
public EdgeTableWrapper(GlobalInstances globals) {
super(globals, globals.getConfig().getEdgeTableName());
}
/**
* Write the given edge to the edge table. Does not
* currently write the edge's properties.
*
* <p/>Note: This only adds the edge information. Vertex
* endpoint information needs to be written to the vertex
* table via {@link VertexTableWrapper}.
* @param edge
*/
public void writeEdge(Edge edge) {
Mutators.apply(getWriter(), new EdgeMutator.Add(edge));
globals.checkedFlush();
}
public void deleteEdge(Edge edge) {
Mutators.apply(getWriter(), new EdgeMutator.Delete(edge));
globals.checkedFlush();
}
public CloseableIterable<Edge> getEdges() {
Scanner scan = getScanner();
scan.fetchColumnFamily(new Text(Constants.LABEL));
if (globals.getConfig().getPreloadedProperties() != null) {
for (String key : globals.getConfig().getPreloadedProperties()) {
scan.fetchColumnFamily(new Text(key));
}
}
final EdgeParser parser = new EdgeParser(globals);
return new ScannerIterable<Edge>(scan) {
@Override
public Edge next(PeekingIterator<Entry<Key, Value>> iterator) {
// TODO could also check local cache before creating a new instance?
String rowId = iterator.peek().getKey().getRow().toString();
List<Entry<Key, Value>> entries =
new ArrayList<Entry<Key, Value>>();
// MDL 05 Jan 2014: Why is this equalsIgnoreCase??
while (iterator.peek() != null && rowId.equalsIgnoreCase(iterator
.peek().getKey().getRow().toString())) {
entries.add(iterator.next());
}
AccumuloEdge edge = parser.parse(rowId, entries);
globals.getCaches().cache(edge, Edge.class);
return edge;
}
};
}
public Iterable<Edge> getEdges(String key, Object value) {
AccumuloGraphUtils.nullCheckProperty(key, value);
if (key.equalsIgnoreCase("label")) {
key = Constants.LABEL;
}
BatchScanner scan = getBatchScanner();
scan.fetchColumnFamily(new Text(key));
byte[] val = AccumuloByteSerializer.serialize(value);
if (val[0] != AccumuloByteSerializer.SERIALIZABLE) {
IteratorSetting is = new IteratorSetting(10, "filter", RegExFilter.class);
RegExFilter.setRegexs(is, null, null, null, Pattern.quote(new String(val)), false);
scan.addScanIterator(is);
return new ScannerIterable<Edge>(scan) {
@Override
public Edge next(PeekingIterator<Entry<Key,Value>> iterator) {
Key k = iterator.next().getKey();
if (k.getColumnFamily().toString().equals(Constants.LABEL)) {
String[] vals = k.getColumnQualifier().toString().split(Constants.ID_DELIM);
return new AccumuloEdge(globals, k.getRow().toString(),
new AccumuloVertex(globals, vals[0]),
new AccumuloVertex(globals, vals[1]), null);
}
return new AccumuloEdge(globals, k.getRow().toString());
}
};
} else {
// TODO
throw new UnsupportedOperationException("Filtering on binary data not currently supported.");
}
}
public void loadEndpointsAndLabel(AccumuloEdge edge) {
Scanner s = getScanner();
try {
s.setRange(new Range(edge.getId().toString()));
s.fetchColumnFamily(new Text(Constants.LABEL));
Iterator<Entry<Key,Value>> iter = s.iterator();
if (!iter.hasNext()) {
dump();
throw new AccumuloGraphException("Unable to find edge row: "+edge);
}
Entry<Key, Value> entry = iter.next();
String cq = entry.getKey().getColumnQualifier().toString();
String[] ids = cq.split(Constants.ID_DELIM);
String label = AccumuloByteSerializer.deserialize(entry.getValue().get());
edge.setVertices(new AccumuloVertex(globals, ids[0]),
new AccumuloVertex(globals, ids[1]));
edge.setLabel(label);
} finally {
s.close();
}
}
}

View File

@@ -0,0 +1,226 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.tables.core;
import java.util.HashSet;
import java.util.Iterator;
import java.util.Map;
import java.util.Set;
import java.util.Map.Entry;
import org.apache.accumulo.core.client.BatchWriter;
import org.apache.accumulo.core.client.IteratorSetting;
import org.apache.accumulo.core.client.Scanner;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Range;
import org.apache.accumulo.core.data.Value;
import org.apache.accumulo.core.iterators.user.RegExFilter;
import org.apache.hadoop.io.Text;
import com.tinkerpop.blueprints.Element;
import com.tinkerpop.blueprints.util.StringFactory;
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
import edu.jhuapl.tinkerpop.Constants;
import edu.jhuapl.tinkerpop.GlobalInstances;
import edu.jhuapl.tinkerpop.mutator.property.ClearPropertyMutator;
import edu.jhuapl.tinkerpop.mutator.property.WritePropertyMutator;
import edu.jhuapl.tinkerpop.mutator.Mutators;
import edu.jhuapl.tinkerpop.parser.PropertyParser;
import edu.jhuapl.tinkerpop.tables.BaseTableWrapper;
/**
* Wrapper around tables with operations
* common to {@link Element}s.
*/
public abstract class ElementTableWrapper extends BaseTableWrapper {
private BatchWriter writer;
public ElementTableWrapper(GlobalInstances globals, String tableName) {
super(globals, tableName);
writer = super.getWriter();
}
/**
* Give a single instance of the writer for this table.
*/
@Override
protected BatchWriter getWriter() {
return writer;
}
/**
* Read the given property from the backing table
* for the given element id.
* @param id
* @param key
* @return
*/
public <V> V readProperty(Element element, String key) {
Scanner s = getScanner();
s.setRange(new Range(element.getId().toString()));
Text colf = StringFactory.LABEL.equals(key)
? new Text(Constants.LABEL) : new Text(key);
s.fetchColumnFamily(colf);
V value = null;
Iterator<Entry<Key, Value>> iter = s.iterator();
if (iter.hasNext()) {
value = AccumuloByteSerializer.deserialize(iter.next().getValue().get());
}
s.close();
return value;
}
/**
* Read all properties for the given element
* from the backing table.
* If the element has no properties, return an empty Map.
* If the element does not exist, return null.
* @param element
* @return
*/
public Map<String, Object> readAllProperties(Element element) {
return readProperties(element, null);
}
/**
* Read the given properties for the given element.
* If propertyKeys is null, read all properties.
* If the element has no properties, return an empty Map.
* If the element does not exist, return null.
* @param id
* @param propertyKeys
* @return
*/
public Map<String, Object> readProperties(Element element, String[] propertyKeys) {
Scanner s = getScanner();
s.setRange(Range.exact((String) element.getId()));
// If propertyKeys is null, we read everything.
// Otherwise, limit to the given attributes.
if (propertyKeys != null) {
s.fetchColumnFamily(new Text(Constants.LABEL));
for (String key : propertyKeys) {
s.fetchColumnFamily(new Text(key));
}
}
Map<String, Object> props = new PropertyParser().parse(s);
s.close();
return props;
}
/**
* Return true if the element with given id exists.
* @param id
* @return
*/
public boolean elementExists(String id) {
Scanner scan = null;
try {
scan = getScanner();
scan.setRange(Range.exact(id));
scan.fetchColumnFamily(new Text(Constants.LABEL));
return new PropertyParser().parse(scan) != null;
} finally {
if (scan != null) {
scan.close();
}
}
}
/**
* Get all property keys for the given element id.
* @param id
* @return
*/
public Set<String> readPropertyKeys(Element element) {
Scanner s = getScanner();
s.setRange(new Range(element.getId().toString()));
Set<String> keys = new HashSet<String>();
for (Entry<Key, Value> entry : s) {
String cf = entry.getKey().getColumnFamily().toString();
keys.add(cf);
}
s.close();
// Remove some special keys.
keys.remove(Constants.IN_EDGE);
keys.remove(Constants.LABEL);
keys.remove(Constants.OUT_EDGE);
return keys;
}
/**
* Delete the property entry from property table.
* @param id
* @param key
*/
public void clearProperty(Element element, String key) {
Mutators.apply(getWriter(),
new ClearPropertyMutator(element.getId().toString(), key));
globals.checkedFlush();
}
/**
* Write the given property to the property table.
* @param id
* @param key
* @param value
*/
public void writeProperty(Element element, String key, Object value) {
Mutators.apply(getWriter(),
new WritePropertyMutator(element.getId().toString(),
key, value));
globals.checkedFlush();
}
/**
* Add custom iterator to the given scanner so that
* it will only return keys with value corresponding to an edge.
* @param scan
* @param labels
*/
protected void applyEdgeLabelValueFilter(Scanner scan, String... labels) {
StringBuilder regex = new StringBuilder();
for (String lab : labels) {
if (regex.length() != 0)
regex.append("|");
regex.append(".*"+Constants.ID_DELIM+"\\Q").append(lab).append("\\E$");
}
IteratorSetting is = new IteratorSetting(10, "edgeValueFilter", RegExFilter.class);
RegExFilter.setRegexs(is, null, null, null, regex.toString(), false);
scan.addScanIterator(is);
}
public void close() {
// TODO?
}
}

View File

@@ -0,0 +1,269 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.tables.core;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Map.Entry;
import java.util.regex.Pattern;
import org.apache.accumulo.core.client.BatchDeleter;
import org.apache.accumulo.core.client.BatchScanner;
import org.apache.accumulo.core.client.IteratorSetting;
import org.apache.accumulo.core.client.Scanner;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Range;
import org.apache.accumulo.core.data.Value;
import org.apache.accumulo.core.iterators.user.RegExFilter;
import org.apache.accumulo.core.util.PeekingIterator;
import org.apache.hadoop.io.Text;
import com.tinkerpop.blueprints.CloseableIterable;
import com.tinkerpop.blueprints.Direction;
import com.tinkerpop.blueprints.Edge;
import com.tinkerpop.blueprints.Vertex;
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
import edu.jhuapl.tinkerpop.AccumuloEdge;
import edu.jhuapl.tinkerpop.AccumuloElement;
import edu.jhuapl.tinkerpop.AccumuloGraphException;
import edu.jhuapl.tinkerpop.AccumuloGraphUtils;
import edu.jhuapl.tinkerpop.AccumuloVertex;
import edu.jhuapl.tinkerpop.Constants;
import edu.jhuapl.tinkerpop.GlobalInstances;
import edu.jhuapl.tinkerpop.ScannerIterable;
import edu.jhuapl.tinkerpop.mutator.vertex.AddVertexMutator;
import edu.jhuapl.tinkerpop.mutator.Mutator;
import edu.jhuapl.tinkerpop.mutator.Mutators;
import edu.jhuapl.tinkerpop.mutator.edge.EdgeEndpointsMutator;
import edu.jhuapl.tinkerpop.parser.VertexParser;
/**
* Wrapper around {@link Vertex} tables.
*/
public class VertexTableWrapper extends ElementTableWrapper {
public VertexTableWrapper(GlobalInstances globals) {
super(globals, globals.getConfig().getVertexTableName());
}
/**
* Write a vertex with the given id.
* Note: This does not currently write the vertex's properties.
* @param vertex
*/
public void writeVertex(Vertex vertex) {
Mutators.apply(getWriter(), new AddVertexMutator(vertex.getId().toString()));
globals.checkedFlush();
}
/**
* Remove the given vertex.
* Note: This uses a BatchDeleter rather than {@link Mutator}
* because it is more efficient.
* @param vertex
*/
public void deleteVertex(Vertex vertex) {
BatchDeleter deleter = null;
try {
deleter = getDeleter();
deleter.setRanges(Arrays.asList(Range.exact((String) vertex.getId())));
deleter.delete();
} catch (Exception e) {
throw new AccumuloGraphException(e);
} finally {
if (deleter != null) {
deleter.close();
}
}
}
/**
* Write edge endpoint information to the vertex table.
* @param edge
*/
public void writeEdgeEndpoints(Edge edge) {
Mutators.apply(getWriter(), new EdgeEndpointsMutator.Add(edge));
globals.checkedFlush();
}
public void deleteEdgeEndpoints(Edge edge) {
Mutators.apply(getWriter(), new EdgeEndpointsMutator.Delete(edge));
globals.checkedFlush();
}
public CloseableIterable<Edge> getEdges(Vertex vertex, Direction direction,
String... labels) {
Scanner scan = getScanner();
scan.setRange(new Range(vertex.getId().toString()));
if (direction.equals(Direction.IN)) {
scan.fetchColumnFamily(new Text(Constants.IN_EDGE));
} else if (direction.equals(Direction.OUT)) {
scan.fetchColumnFamily(new Text(Constants.OUT_EDGE));
} else {
scan.fetchColumnFamily(new Text(Constants.IN_EDGE));
scan.fetchColumnFamily(new Text(Constants.OUT_EDGE));
}
if (labels.length > 0) {
applyEdgeLabelValueFilter(scan, labels);
}
return new ScannerIterable<Edge>(scan) {
@Override
public Edge next(PeekingIterator<Entry<Key,Value>> iterator) {
// TODO better use of information readily available...
// TODO could also check local cache before creating a new
// instance?
Entry<Key,Value> kv = iterator.next();
String[] parts = kv.getKey().getColumnQualifier().toString().split(Constants.ID_DELIM);
String label = (new String(kv.getValue().get())).split(Constants.ID_DELIM)[1];
AccumuloEdge edge;
if (kv.getKey().getColumnFamily().toString().equalsIgnoreCase(Constants.IN_EDGE)) {
edge = new AccumuloEdge(globals, parts[1],
new AccumuloVertex(globals, kv.getKey().getRow().toString()),
new AccumuloVertex(globals, parts[0]), label);
} else {
edge = new AccumuloEdge(globals, parts[1],
new AccumuloVertex(globals, parts[0]),
new AccumuloVertex(globals, kv.getKey().getRow().toString()), label);
}
globals.getCaches().cache(edge, Edge.class);
return edge;
}
};
}
public Iterable<Vertex> getVertices(Vertex vertex, Direction direction, String... labels) {
Scanner scan = getScanner();
scan.setRange(new Range(vertex.getId().toString()));
if (direction.equals(Direction.IN)) {
scan.fetchColumnFamily(new Text(Constants.IN_EDGE));
} else if (direction.equals(Direction.OUT)) {
scan.fetchColumnFamily(new Text(Constants.OUT_EDGE));
} else {
scan.fetchColumnFamily(new Text(Constants.IN_EDGE));
scan.fetchColumnFamily(new Text(Constants.OUT_EDGE));
}
if (labels != null && labels.length > 0) {
applyEdgeLabelValueFilter(scan, labels);
}
return new ScannerIterable<Vertex>(scan) {
@Override
public Vertex next(PeekingIterator<Entry<Key,Value>> iterator) {
// TODO better use of information readily available...
// TODO could also check local cache before creating a new
// instance?
String[] parts = iterator.next().getKey().getColumnQualifier()
.toString().split(Constants.ID_DELIM);
AccumuloVertex vertex = new AccumuloVertex(globals, parts[0]);
globals.getCaches().cache(vertex, Vertex.class);
return vertex;
}
};
}
public CloseableIterable<Vertex> getVertices() {
return getVerticesInRange(null, null);
}
public CloseableIterable<Vertex> getVerticesInRange(Object fromId, Object toId) {
Scanner scan = getScanner();
scan.setRange(new Range(fromId != null ? fromId.toString() : null,
toId != null ? toId.toString() : null));
scan.fetchColumnFamily(new Text(Constants.LABEL));
if (globals.getConfig().getPreloadedProperties() != null) {
for (String key : globals.getConfig().getPreloadedProperties()) {
scan.fetchColumnFamily(new Text(key));
}
}
final VertexParser parser = new VertexParser(globals);
return new ScannerIterable<Vertex>(scan) {
@Override
public Vertex next(PeekingIterator<Entry<Key, Value>> iterator) {
// TODO could also check local cache before creating a new instance?
String rowId = iterator.peek().getKey().getRow().toString();
List<Entry<Key, Value>> entries =
new ArrayList<Entry<Key, Value>>();
while (iterator.peek() != null && rowId.equals(iterator
.peek().getKey().getRow().toString())) {
entries.add(iterator.next());
}
AccumuloVertex vertex = parser.parse(rowId, entries);
globals.getCaches().cache(vertex, Vertex.class);
return vertex;
}
};
}
public Iterable<Vertex> getVertices(String key, Object value) {
AccumuloGraphUtils.validateProperty(key, value);
byte[] val = AccumuloByteSerializer.serialize(value);
if (val[0] != AccumuloByteSerializer.SERIALIZABLE) {
BatchScanner scan = getBatchScanner();
scan.fetchColumnFamily(new Text(key));
IteratorSetting is = new IteratorSetting(10, "filter", RegExFilter.class);
RegExFilter.setRegexs(is, null, null, null, Pattern.quote(new String(val)), false);
scan.addScanIterator(is);
return new ScannerIterable<Vertex>(scan) {
@Override
public Vertex next(PeekingIterator<Entry<Key,Value>> iterator) {
Entry<Key, Value> kv = iterator.next();
String key = kv.getKey().getColumnFamily().toString();
Object value = AccumuloByteSerializer.deserialize(kv.getValue().get());
Vertex v = globals.getCaches().retrieve(kv.getKey().getRow().toString(), Vertex.class);
if (v == null) {
v = new AccumuloVertex(globals, kv.getKey().getRow().toString());
}
((AccumuloElement) v).setPropertyInMemory(key, value);
globals.getCaches().cache(v, Vertex.class);
return v;
}
};
} else {
// TODO
throw new UnsupportedOperationException("Filtering on binary data not currently supported.");
}
}
}

View File

@@ -0,0 +1,179 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.tables.index;
import java.util.Arrays;
import java.util.Collections;
import java.util.Map.Entry;
import java.util.regex.Pattern;
import org.apache.accumulo.core.client.BatchDeleter;
import org.apache.accumulo.core.client.BatchWriter;
import org.apache.accumulo.core.client.IteratorSetting;
import org.apache.accumulo.core.client.Scanner;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Range;
import org.apache.accumulo.core.data.Value;
import org.apache.accumulo.core.iterators.user.RegExFilter;
import org.apache.accumulo.core.util.PeekingIterator;
import org.apache.hadoop.io.Text;
import com.tinkerpop.blueprints.CloseableIterable;
import com.tinkerpop.blueprints.Element;
import com.tinkerpop.blueprints.IndexableGraph;
import com.tinkerpop.blueprints.Vertex;
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
import edu.jhuapl.tinkerpop.AccumuloElement;
import edu.jhuapl.tinkerpop.AccumuloGraphException;
import edu.jhuapl.tinkerpop.AccumuloGraphUtils;
import edu.jhuapl.tinkerpop.GlobalInstances;
import edu.jhuapl.tinkerpop.ScannerIterable;
import edu.jhuapl.tinkerpop.mutator.Mutators;
import edu.jhuapl.tinkerpop.mutator.index.IndexValueMutator;
import edu.jhuapl.tinkerpop.parser.EdgeIndexParser;
import edu.jhuapl.tinkerpop.parser.ElementIndexParser;
import edu.jhuapl.tinkerpop.parser.VertexIndexParser;
import edu.jhuapl.tinkerpop.tables.BaseTableWrapper;
/**
* Wrapper around index tables containing properties
* and values.
*/
public abstract class BaseIndexValuesTableWrapper extends BaseTableWrapper {
protected final Class<? extends Element> elementType;
protected BaseIndexValuesTableWrapper(GlobalInstances globals,
Class<? extends Element> elementType, String tableName) {
super(globals, tableName);
this.elementType = elementType;
}
/**
* Return class of this index.
* @return
*/
public Class<? extends Element> getElementType() {
return elementType;
}
/**
* Add the property to this index, if autoindexing is enabled
* and/or the given key has indexing enabled.
* @param element
* @param key
* @param value
*/
public void setPropertyForIndex(Element element, String key, Object value) {
setPropertyForIndex(element, key, value, false);
}
/**
* Add the property to this index.
*
* <p/>Note that this requires a round-trip to Accumulo to see
* if the property exists if the provided key has an index.
* So for best performance, create indices after bulk ingest.
* <p/>If the force parameter is true, set the property regardless
* of whether indexing is enabled for the given key. This is needed
* for {@link IndexableGraph} operations.
* @param element
* @param key
* @param value
* @param force
*/
public void setPropertyForIndex(Element element, String key, Object value,
boolean force) {
AccumuloGraphUtils.validateProperty(key, value);
if (force || globals.getConfig().getAutoIndex() ||
globals.getIndexMetadataWrapper()
.getIndexedKeys(elementType).contains(key)) {
BatchWriter writer = getWriter();
Object oldValue = element.getProperty(key);
if (oldValue != null && !oldValue.equals(value)) {
Mutators.apply(writer, new IndexValueMutator.Delete(element, key, oldValue));
}
Mutators.apply(writer, new IndexValueMutator.Add(element, key, value));
globals.checkedFlush();
}
}
/**
* Remove property from the index.
* @param element
* @param key
* @param value
*/
public void removePropertyFromIndex(Element element, String key, Object value) {
if (value != null) {
Mutators.apply(getWriter(), new IndexValueMutator.Delete(element, key, value));
globals.checkedFlush();
}
}
/**
* Get elements with the key/value pair.
* @param key
* @param value
* @return
*/
@SuppressWarnings("unchecked")
public <T extends Element> CloseableIterable<T> readElementsFromIndex(String key, Object value) {
Scanner scan = getScanner();
byte[] id = AccumuloByteSerializer.serialize(value);
scan.setRange(Range.exact(new Text(id)));
scan.fetchColumnFamily(new Text(key));
final ElementIndexParser<? extends AccumuloElement> parser =
Vertex.class.equals(elementType) ? new VertexIndexParser(globals) :
new EdgeIndexParser(globals);
return new ScannerIterable<T>(scan) {
@Override
public T next(PeekingIterator<Entry<Key,Value>> iterator) {
return (T) parser.parse(Arrays.asList(iterator.next()));
}
};
}
/**
* Remove the given element's properties from the index.
* @param element
*/
public void removeElementFromIndex(Element element) {
BatchDeleter deleter = null;
try {
deleter = getDeleter();
deleter.setRanges(Collections.singleton(new Range()));
IteratorSetting is = new IteratorSetting(10, "getEdgeFilter", RegExFilter.class);
RegExFilter.setRegexs(is, null, null,
"^"+Pattern.quote(element.getId().toString())+"$", null, false);
deleter.addScanIterator(is);
deleter.delete();
deleter.close();
} catch (Exception e) {
throw new AccumuloGraphException(e);
} finally {
if (deleter != null) {
deleter.close();
}
}
}
}

View File

@@ -0,0 +1,77 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.tables.index;
import com.tinkerpop.blueprints.CloseableIterable;
import com.tinkerpop.blueprints.Edge;
import com.tinkerpop.blueprints.Element;
import com.tinkerpop.blueprints.Vertex;
import edu.jhuapl.tinkerpop.AccumuloGraphException;
import edu.jhuapl.tinkerpop.GlobalInstances;
import edu.jhuapl.tinkerpop.tables.core.EdgeTableWrapper;
import edu.jhuapl.tinkerpop.tables.core.ElementTableWrapper;
import edu.jhuapl.tinkerpop.tables.core.VertexTableWrapper;
/**
* Base class for key index tables.
*/
public abstract class BaseKeyIndexTableWrapper extends BaseIndexValuesTableWrapper {
protected BaseKeyIndexTableWrapper(GlobalInstances globals,
Class<? extends Element> elementType, String tableName) {
super(globals, elementType, tableName);
}
/**
* Rebuild this index for the given table.
* @param table
* @param key
*/
public void rebuildIndex(String key, Class<? extends Element> elementClass) {
ElementTableWrapper wrapper = globals.getElementWrapper(elementClass);
if (wrapper instanceof VertexTableWrapper) {
CloseableIterable<Vertex> iter = ((VertexTableWrapper) wrapper).getVertices();
for (Vertex v : iter) {
rebuild(wrapper, v, key);
}
iter.close();
}
else if (wrapper instanceof EdgeTableWrapper) {
CloseableIterable<Edge> iter = ((EdgeTableWrapper) wrapper).getEdges();
for (Edge e : iter) {
rebuild(wrapper, e, key);
}
iter.close();
}
else {
throw new AccumuloGraphException("Unexpected table wrapper: "+wrapper.getClass());
}
globals.checkedFlush();
}
/**
* Add given element to index for the given key.
* @param element
* @param key
*/
private void rebuild(ElementTableWrapper wrapper,
Element element, String key) {
Object value = wrapper.readProperty(element, key);
if (value != null) {
setPropertyForIndex(element, key, value);
}
}
}

View File

@@ -0,0 +1,86 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.tables.index;
import java.util.Arrays;
import java.util.Map.Entry;
import org.apache.accumulo.core.client.Scanner;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Range;
import org.apache.accumulo.core.data.Value;
import org.apache.accumulo.core.util.PeekingIterator;
import org.apache.hadoop.io.Text;
import com.tinkerpop.blueprints.Edge;
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
import edu.jhuapl.tinkerpop.AccumuloEdge;
import edu.jhuapl.tinkerpop.GlobalInstances;
import edu.jhuapl.tinkerpop.ScannerIterable;
import edu.jhuapl.tinkerpop.parser.EdgeIndexParser;
/**
* Wrapper around {@link Edge} index table.
*/
public class EdgeKeyIndexTableWrapper extends BaseKeyIndexTableWrapper {
public EdgeKeyIndexTableWrapper(GlobalInstances globals) {
super(globals, Edge.class, globals.getConfig()
.getEdgeKeyIndexTableName());
}
/**
* Retrieve edges from the index table based
* on the given key/value.
* @param key
* @param value
* @return
*/
public Iterable<Edge> getEdges(String key, Object value) {
Scanner s = getScanner();
Text row = new Text(AccumuloByteSerializer.serialize(value));
s.setRange(Range.exact(row));
s.fetchColumnFamily(new Text(key));
final EdgeIndexParser parser = new EdgeIndexParser(globals);
return new ScannerIterable<Edge>(s) {
@Override
public Edge next(PeekingIterator<Entry<Key, Value>> iterator) {
Entry<Key, Value> entry = iterator.next();
AccumuloEdge e = parser.parse(Arrays.asList(entry));
// Check if we have it cached already, in which
// case use the cached version.
AccumuloEdge cached = (AccumuloEdge) globals.getCaches()
.retrieve(e.getId(), Edge.class);
if (cached != null) {
for (String key : e.getPropertyKeysInMemory()) {
cached.setPropertyInMemory(key, e.getPropertyInMemory(key));
}
return cached;
}
// We don't have it, so cache the new one and return it.
globals.getCaches().cache(e, Edge.class);
return e;
}
};
}
}

View File

@@ -0,0 +1,170 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.tables.index;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import org.apache.accumulo.core.client.Scanner;
import org.apache.hadoop.io.Text;
import com.tinkerpop.blueprints.Element;
import com.tinkerpop.blueprints.Index;
import com.tinkerpop.blueprints.IndexableGraph;
import com.tinkerpop.blueprints.KeyIndexableGraph;
import com.tinkerpop.blueprints.util.ExceptionFactory;
import edu.jhuapl.tinkerpop.AccumuloIndex;
import edu.jhuapl.tinkerpop.Constants.IndexMetadataEntryType;
import edu.jhuapl.tinkerpop.GlobalInstances;
import edu.jhuapl.tinkerpop.mutator.Mutators;
import edu.jhuapl.tinkerpop.mutator.index.IndexMetadataMutator;
import edu.jhuapl.tinkerpop.parser.IndexedItem;
import edu.jhuapl.tinkerpop.parser.IndexedItemsListParser;
import edu.jhuapl.tinkerpop.tables.BaseTableWrapper;
/**
* Stores metadata, in particular the indexed keys
* for {@link KeyIndexableGraph}, and the list of
* named indexes for {@link IndexableGraph}.
*/
public class IndexMetadataTableWrapper extends BaseTableWrapper {
public IndexMetadataTableWrapper(GlobalInstances globals) {
super(globals, globals.getConfig().getIndexMetadataTableName());
}
//////// Methods for KeyIndexableGraph ////////
public void writeKeyMetadataEntry(String key, Class<? extends Element> clazz) {
Mutators.apply(getWriter(), new IndexMetadataMutator.Add(key, clazz,
IndexMetadataEntryType.__INDEX_KEY__));
}
public void clearKeyMetadataEntry(String key, Class<? extends Element> clazz) {
Mutators.apply(getWriter(), new IndexMetadataMutator.Delete(key, clazz,
IndexMetadataEntryType.__INDEX_KEY__));
}
public <T extends Element> Set<String> getIndexedKeys(Class<T> elementClass) {
if (elementClass == null) {
throw ExceptionFactory.classForElementCannotBeNull();
}
IndexedItemsListParser parser = new IndexedItemsListParser(elementClass);
Scanner scan = null;
try {
scan = getScanner();
scan.fetchColumnFamily(new Text(IndexMetadataEntryType.__INDEX_KEY__.name()));
Set<String> keys = new HashSet<String>();
for (IndexedItem item : parser.parse(scan)) {
keys.add(item.getKey());
}
return keys;
} finally {
if (scan != null) {
scan.close();
}
}
}
//////// Methods for IndexableGraph ////////
@SuppressWarnings({"rawtypes", "unchecked"})
public Iterable<Index<? extends Element>> getIndices() {
List<Index<? extends Element>> indexes = new ArrayList<Index<? extends Element>>();
IndexedItemsListParser parser = new IndexedItemsListParser();
Scanner scan = null;
try {
scan = getScanner();
scan.fetchColumnFamily(new Text(IndexMetadataEntryType.__INDEX_NAME__.name()));
for (IndexedItem item : parser.parse(scan)) {
indexes.add(new AccumuloIndex(globals,
item.getKey(), item.getElementClass()));
}
return indexes;
} finally {
if (scan != null) {
scan.close();
}
}
}
public <T extends Element> Index<T> getIndex(String indexName,
Class<T> indexClass) {
IndexedItemsListParser parser = new IndexedItemsListParser();
Scanner scan = null;
try {
scan = getScanner();
scan.fetchColumnFamily(new Text(IndexMetadataEntryType.__INDEX_NAME__.name()));
for (IndexedItem item : parser.parse(scan)) {
if (item.getKey().equals(indexName)) {
if (item.getElementClass().equals(indexClass)) {
return new AccumuloIndex<T>(globals, indexName,
indexClass);
}
else {
throw ExceptionFactory.indexDoesNotSupportClass(indexName, indexClass);
}
}
}
return null;
} finally {
if (scan != null) {
scan.close();
}
}
}
public <T extends Element> Index<T> createIndex(String indexName,
Class<T> indexClass) {
for (Index<?> index : getIndices()) {
if (index.getIndexName().equals(indexName)) {
throw ExceptionFactory.indexAlreadyExists(indexName);
}
}
writeIndexNameEntry(indexName, indexClass);
return new AccumuloIndex<T>(globals, indexName, indexClass);
}
private void writeIndexNameEntry(String indexName,
Class<? extends Element> indexClass) {
Mutators.apply(getWriter(), new IndexMetadataMutator.Add(indexName,
indexClass, IndexMetadataEntryType.__INDEX_NAME__));
}
public void clearIndexNameEntry(String indexName,
Class<? extends Element> indexClass) {
Mutators.apply(getWriter(), new IndexMetadataMutator.Delete(indexName,
indexClass, IndexMetadataEntryType.__INDEX_NAME__));
}
}

View File

@@ -0,0 +1,32 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.tables.index;
import com.tinkerpop.blueprints.Element;
import com.tinkerpop.blueprints.IndexableGraph;
import edu.jhuapl.tinkerpop.GlobalInstances;
/**
* Wrapper around a named index table (for {@link IndexableGraph}).
*/
public class NamedIndexTableWrapper extends BaseIndexValuesTableWrapper {
public NamedIndexTableWrapper(GlobalInstances globals,
Class<? extends Element> elementType, String indexName) {
super(globals, elementType,
globals.getConfig().getNamedIndexTableName(indexName));
}
}

View File

@@ -0,0 +1,85 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop.tables.index;
import java.util.Arrays;
import java.util.Map.Entry;
import org.apache.accumulo.core.client.Scanner;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Range;
import org.apache.accumulo.core.data.Value;
import org.apache.accumulo.core.util.PeekingIterator;
import org.apache.hadoop.io.Text;
import com.tinkerpop.blueprints.Vertex;
import edu.jhuapl.tinkerpop.AccumuloByteSerializer;
import edu.jhuapl.tinkerpop.AccumuloVertex;
import edu.jhuapl.tinkerpop.GlobalInstances;
import edu.jhuapl.tinkerpop.ScannerIterable;
import edu.jhuapl.tinkerpop.parser.VertexIndexParser;
/**
* Wrapper around {@link Vertex} index table.
*/
public class VertexKeyIndexTableWrapper extends BaseKeyIndexTableWrapper {
public VertexKeyIndexTableWrapper(GlobalInstances globals) {
super(globals, Vertex.class, globals.getConfig()
.getVertexKeyIndexTableName());
}
/**
* Use the index to retrieve vertices with the
* given key/value.
* @param key
* @param value
*/
public Iterable<Vertex> getVertices(String key, Object value) {
Scanner s = getScanner();
Text row = new Text(AccumuloByteSerializer.serialize(value));
s.setRange(Range.exact(row));
s.fetchColumnFamily(new Text(key));
final VertexIndexParser parser = new VertexIndexParser(globals);
return new ScannerIterable<Vertex>(s) {
@Override
public Vertex next(PeekingIterator<Entry<Key, Value>> iterator) {
Entry<Key, Value> entry = iterator.next();
AccumuloVertex v = parser.parse(Arrays.asList(entry));
// Check if we have it cached already, in which
// case use the cached version.
AccumuloVertex cached = (AccumuloVertex) globals.getCaches()
.retrieve(v.getId(), Vertex.class);
if (cached != null) {
for (String key : v.getPropertyKeysInMemory()) {
cached.setPropertyInMemory(key, v.getPropertyInMemory(key));
}
return cached;
}
// We don't have it, so cache the new one and return it.
globals.getCaches().cache(v, Vertex.class);
return v;
}
};
}
}

View File

@@ -0,0 +1,6 @@
log4j.rootLogger=INFO, stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c - %m%n

View File

@@ -0,0 +1,64 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertTrue;
import java.util.Iterator;
import org.junit.Test;
import com.tinkerpop.blueprints.Direction;
import com.tinkerpop.blueprints.Edge;
import com.tinkerpop.blueprints.Vertex;
public class AccumuloBulkIngesterTest {
@Test
public void testBulkIngester() throws Exception {
AccumuloGraphConfiguration cfg = AccumuloGraphTestUtils.generateGraphConfig("propertyBuilder").setClear(true);
AccumuloBulkIngester ingester = new AccumuloBulkIngester(cfg);
for (String t : cfg.getTableNames()) {
assertTrue(cfg.getConnector().tableOperations().exists(t));
}
ingester.addVertex("A").finish();
ingester.addVertex("B").add("P1", "V1").add("P2", "2").finish();
ingester.addEdge("A", "B", "edge").add("P3", "V3").finish();
ingester.shutdown(true);
AccumuloGraph graph = new AccumuloGraph(cfg.clone().setClear(false));
Vertex v1 = graph.getVertex("A");
assertNotNull(v1);
Iterator<Edge> it = v1.getEdges(Direction.OUT).iterator();
assertTrue(it.hasNext());
Edge e = it.next();
assertEquals("edge", e.getLabel());
Vertex v2 = e.getVertex(Direction.IN);
assertEquals("B", v2.getId());
assertEquals("V1", v2.getProperty("P1"));
assertEquals("2", v2.getProperty("P2"));
graph.shutdown();
}
}

View File

@@ -0,0 +1,119 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop;
import static org.junit.Assert.*;
import org.junit.Test;
import com.tinkerpop.blueprints.Edge;
import com.tinkerpop.blueprints.Graph;
import com.tinkerpop.blueprints.Vertex;
/**
* Tests related to Accumulo elements.
*/
public class AccumuloElementTest {
@Test
public void testNonStringIds() throws Exception {
Graph graph = AccumuloGraphTestUtils.makeGraph("nonStringIds");
Object[] ids = new Object[] {
10, 20, 30L, 40L,
50.0f, 60.0f, 70.0d, 80.0d,
(byte) 'a', (byte) 'b', 'c', 'd',
"str1", "str2",
new GenericObject("str3"), new GenericObject("str4"),
};
Object[] edgeIds = new Object[] {
100, 200, 300L, 400L,
500.0f, 600.0f, 700.0d, 800.0d,
(byte) 'e', (byte) 'f', 'g', 'h',
"str5", "str6",
new GenericObject("str7"), new GenericObject("str8"),
};
for (int i = 0; i < ids.length; i++) {
assertNull(graph.getVertex(ids[i]));
Vertex v = graph.addVertex(ids[i]);
assertNotNull(v);
assertNotNull(graph.getVertex(ids[i]));
}
assertEquals(ids.length, count(graph.getVertices()));
for (int i = 1; i < edgeIds.length; i++) {
assertNull(graph.getEdge(edgeIds[i-1]));
Edge e = graph.addEdge(edgeIds[i-1],
graph.getVertex(ids[i-1]),
graph.getVertex(ids[i]), "label");
assertNotNull(e);
assertNotNull(graph.getEdge(edgeIds[i-1]));
}
assertEquals(edgeIds.length-1, count(graph.getEdges()));
graph.shutdown();
}
@Test
public void testGetVerticesInRange() {
AccumuloGraph graph = (AccumuloGraph) AccumuloGraphTestUtils
.makeGraph("testGetVerticesInRange");
assertEquals(0, count(graph.getVerticesInRange(null, null)));
for (int i = 0; i < 10; i++) {
graph.addVertex(id(i));
}
assertEquals(10, count(graph.getVerticesInRange(null, null)));
assertEquals(0, count(graph.getVerticesInRange(id(20), null)));
assertEquals(10, count(graph.getVerticesInRange(null, id(20))));
assertEquals(0, count(graph.getVerticesInRange(id(20), id(20))));
for (int i = 0; i < 10; i++) {
assertEquals(i+1, count(graph.getVerticesInRange(null, id(i))));
assertEquals(10-i, count(graph.getVerticesInRange(id(i), null)));
assertEquals(1, count(graph.getVerticesInRange(id(i), id(i))));
}
graph.shutdown();
}
private static String id(int idNum) {
return String.format("%08d", idNum);
}
private static int count(Iterable<?> iter) {
int count = 0;
for (@SuppressWarnings("unused") Object obj : iter) {
count++;
}
return count;
}
private static class GenericObject {
private final Object id;
public GenericObject(Object id) {
this.id = id;
}
@Override
public String toString() {
return "GenericObject [id=" + id + "]";
}
}
}

View File

@@ -18,21 +18,36 @@ import static org.junit.Assert.*;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Iterator;
import java.util.List;
import javax.xml.namespace.QName;
import org.apache.commons.configuration.Configuration;
import org.apache.hadoop.io.Text;
import org.junit.Test;
import com.tinkerpop.blueprints.Direction;
import com.tinkerpop.blueprints.Edge;
import com.tinkerpop.blueprints.GraphFactory;
import com.tinkerpop.blueprints.Vertex;
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration.InstanceType;
public class AccumuloGraphConfigurationTest {
@Test
public void testConfigurationInterface() throws Exception {
Configuration conf = AccumuloGraphTestUtils.generateGraphConfig("setPropsValid");
for (String key : AccumuloGraphConfiguration.getValidInternalKeys()) {
// This is bad... but we should allow them if they are valid keys.
conf.setProperty(key, "value");
}
conf = AccumuloGraphTestUtils.generateGraphConfig("setPropsInvalid");
try {
conf.setProperty("invalidKey", "value");
fail();
} catch (Exception e) { }
}
@Test
public void testSplits() throws Exception {
AccumuloGraphConfiguration cfg;
@@ -136,49 +151,146 @@ public class AccumuloGraphConfigurationTest {
}
graph.shutdown();
graph = new AccumuloGraph(cfg.setCreate(false));
graph = new AccumuloGraph(cfg.clone().setCreate(false));
assertTrue(graph.isEmpty());
graph.addVertex("A");
graph.addVertex("B");
assertFalse(graph.isEmpty());
graph.shutdown();
graph = new AccumuloGraph(cfg.setClear(true));
graph = new AccumuloGraph(cfg.clone().setClear(true));
assertTrue(graph.isEmpty());
graph.shutdown();
}
@Test
public void testBulkIngester() throws Exception {
AccumuloGraphConfiguration cfg = AccumuloGraphTestUtils.generateGraphConfig("propertyBuilder").setClear(true);
public void testPrint() throws Exception {
AccumuloGraphConfiguration cfg =
AccumuloGraphTestUtils.generateGraphConfig("printTest");
cfg.print();
}
AccumuloBulkIngester ingester = new AccumuloBulkIngester(cfg);
@Test
public void testInvalidCacheParams() throws Exception {
int size = 100;
int timeout = 30000;
for (String t : cfg.getTableNames()) {
assertTrue(cfg.getConnector().tableOperations().exists(t));
AccumuloGraphConfiguration cfg = AccumuloGraphTestUtils
.generateGraphConfig("cacheParams");
cfg.validate();
// Vertex cache.
assertFalse(cfg.getVertexCacheEnabled());
try {
cfg.setVertexCacheParams(-1, timeout);
fail();
} catch (Exception e) { }
try {
cfg.setVertexCacheParams(size, -1);
fail();
} catch (Exception e) { }
assertFalse(cfg.getVertexCacheEnabled());
cfg.setVertexCacheParams(size, timeout);
cfg.validate();
assertTrue(cfg.getVertexCacheEnabled());
assertEquals(size, cfg.getVertexCacheSize());
assertEquals(timeout, cfg.getVertexCacheTimeout());
cfg.setVertexCacheParams(-1, -1);
cfg.validate();
assertFalse(cfg.getVertexCacheEnabled());
// Edge cache.
assertFalse(cfg.getEdgeCacheEnabled());
try {
cfg.setEdgeCacheParams(-1, timeout);
fail();
} catch (Exception e) { }
try {
cfg.setEdgeCacheParams(size, -1);
fail();
} catch (Exception e) { }
assertFalse(cfg.getEdgeCacheEnabled());
cfg.setEdgeCacheParams(size, timeout);
cfg.validate();
assertTrue(cfg.getEdgeCacheEnabled());
assertEquals(size, cfg.getEdgeCacheSize());
assertEquals(timeout, cfg.getEdgeCacheTimeout());
cfg.setEdgeCacheParams(-1, -1);
cfg.validate();
assertFalse(cfg.getEdgeCacheEnabled());
}
/**
* Test different kinds of graph names (hyphens, punctuation, etc).
* @throws Exception
*/
@Test
public void testGraphNames() throws Exception {
AccumuloGraphConfiguration conf = new AccumuloGraphConfiguration();
String[] valid = new String[] {
"alpha", "12345", "alnum12345",
"12345alnum", "under_score1", "_under_score_2"};
String[] invalid = new String[] {"hyph-en",
"dot..s", "quo\"tes"};
for (String name : valid) {
conf.setGraphName(name);
}
ingester.addVertex("A").finish();
ingester.addVertex("B").add("P1", "V1").add("P2", "2").finish();
ingester.addEdge("A", "B", "edge").add("P3", "V3").finish();
ingester.shutdown(true);
for (String name : invalid) {
try {
conf.setGraphName(name);
fail();
} catch (Exception e) { }
}
}
cfg.setClear(false);
AccumuloGraph graph = new AccumuloGraph(cfg);
Vertex v1 = graph.getVertex("A");
assertNotNull(v1);
@Test
public void testPreloadedProperties() {
// Don't allow "all" and "some" preloaded properties.
AccumuloGraphConfiguration conf = new AccumuloGraphConfiguration();
conf.setPreloadAllProperties(true);
conf.setPreloadedProperties(new String[]{"one", "two", "three"});
try {
conf.validate();
fail();
} catch (Exception e) { }
}
Iterator<Edge> it = v1.getEdges(Direction.OUT).iterator();
assertTrue(it.hasNext());
@Test
public void testImmutableConnector() throws Exception {
AccumuloGraphConfiguration cfg = new AccumuloGraphConfiguration().setInstanceType(
InstanceType.Mock).setGraphName("immutableConnector")
.setCreate(true).setAutoFlush(false);
Edge e = it.next();
assertEquals("edge", e.getLabel());
cfg.getConnector();
Vertex v2 = e.getVertex(Direction.IN);
assertEquals("B", v2.getId());
assertEquals("V1", v2.getProperty("P1"));
assertEquals("2", v2.getProperty("P2"));
try {
cfg.setCreate(false);
fail();
} catch (Exception e) { }
graph.shutdown();
try {
cfg.setAutoFlush(true);
fail();
} catch (Exception e) { }
assertTrue(cfg.getCreate());
assertFalse(cfg.getAutoFlush());
}
}

View File

@@ -88,6 +88,7 @@ public class AccumuloGraphTest extends GraphTest {
printTestPerformance("GraphSONReaderTestSuite", this.stopWatch());
}
@Override
public void doTestSuite(final TestSuite testSuite) throws Exception {
String doTest = System.getProperty("testTinkerGraph");
if (doTest == null || doTest.equals("true")) {
@@ -112,12 +113,21 @@ public class AccumuloGraphTest extends GraphTest {
new AccumuloGraphTest().generateGraph();
}
@Override
public void dropGraph(final String graphDirectoryName) {
if (graphDirectoryName != null) {
((AccumuloGraph) generateGraph(graphDirectoryName)).clear();
AccumuloGraphConfiguration cfg = AccumuloGraphTestUtils.generateGraphConfig(graphDirectoryName);
try {
for (String table : cfg.getConnector().tableOperations().list()) {
cfg.getConnector().tableOperations().delete(table);
}
} catch (Exception e) {
throw new AccumuloGraphException(e);
}
}
}
@Override
public Object convertId(final Object id) {
return id.toString();
}

View File

@@ -14,15 +14,19 @@
*/
package edu.jhuapl.tinkerpop;
import com.tinkerpop.blueprints.Graph;
import com.tinkerpop.blueprints.GraphFactory;
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration.InstanceType;
public class AccumuloGraphTestUtils {
public static AccumuloGraphConfiguration generateGraphConfig(String graphDirectoryName) {
AccumuloGraphConfiguration cfg = new AccumuloGraphConfiguration();
cfg.setInstanceName("instanceName").setZookeeperHosts("ZookeeperHostsString");
cfg.setUser("root").setPassword("".getBytes());
cfg.setGraphName(graphDirectoryName).setCreate(true).autoFlush(true).setInstanceType(InstanceType.Mock);
return cfg;
return new AccumuloGraphConfiguration().setInstanceType(InstanceType.Mock)
.setGraphName(graphDirectoryName).setCreate(true);
}
public static Graph makeGraph(String name) {
return GraphFactory.open(generateGraphConfig(name));
}
}

View File

@@ -2,55 +2,117 @@ package edu.jhuapl.tinkerpop;
import static org.junit.Assert.*;
import java.io.IOException;
import java.util.Map.Entry;
import org.apache.accumulo.core.client.AccumuloException;
import org.apache.accumulo.core.client.AccumuloSecurityException;
import org.apache.accumulo.core.client.Scanner;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Value;
import org.junit.Test;
import com.tinkerpop.blueprints.Edge;
import com.tinkerpop.blueprints.Element;
import com.tinkerpop.blueprints.GraphFactory;
import com.tinkerpop.blueprints.Vertex;
public class AutoIndexTest {
@Test
public void testIndexCreation() throws AccumuloException, AccumuloSecurityException, IOException, InterruptedException {
AccumuloGraph ag = (AccumuloGraph) GraphFactory.open(AccumuloGraphTestUtils.generateGraphConfig("AutoIndexTest").setAutoIndex(true).getConfiguration());
String VERT = "1234";
String KEY = "name";
String VALUE = "bananaman";
public void testVertexAutoIndex() throws Exception {
AccumuloGraph graph = (AccumuloGraph) GraphFactory.open(AccumuloGraphTestUtils
.generateGraphConfig("VertexAutoIndexTest").setAutoIndex(true).getConfiguration());
String id = "1234";
String key = "name";
String value = "bananaman";
Vertex v1 = ag.addVertex(VERT);
v1.setProperty(KEY, VALUE);
Vertex v1 = graph.addVertex(id);
v1.setProperty(key, value);
Scanner scan = ag.getVertexIndexScanner();
for (Entry<Key,Value> kv : scan) {
assertEquals(new String(AccumuloByteSerializer.serialize(VALUE)), kv.getKey().getRow().toString());
assertEquals(KEY, kv.getKey().getColumnFamily().toString());
assertEquals(VERT, kv.getKey().getColumnQualifier().toString());
Iterable<Element> elements = graph.getGlobals()
.getVertexKeyIndexWrapper().readElementsFromIndex(key, value);
int count = 0;
for (Element element : elements) {
assertTrue(element instanceof Vertex);
assertEquals(id, element.getId());
assertEquals(value, element.getProperty(key));
count++;
}
assertEquals(1, count);
graph.removeVertex(v1);
elements = graph.getGlobals()
.getVertexKeyIndexWrapper().readElementsFromIndex(key, value);
assertEquals(0, count(elements));
}
@Test
public void testRegularCreation() throws AccumuloException, AccumuloSecurityException, IOException, InterruptedException {
AccumuloGraph ag = (AccumuloGraph) GraphFactory.open(AccumuloGraphTestUtils.generateGraphConfig("NoAutoIndexTest").getConfiguration());
String VERT = "1234";
String KEY = "name";
String VALUE = "bananaman";
public void testVertexNoAutoIndex() throws Exception {
AccumuloGraph graph = (AccumuloGraph) GraphFactory.open(AccumuloGraphTestUtils
.generateGraphConfig("VertexNoAutoIndexTest").getConfiguration());
String id = "1234";
String key = "name";
String value = "bananaman";
Vertex v1 = ag.addVertex(VERT);
v1.setProperty(KEY, VALUE);
Scanner scan = ag.getVertexIndexScanner();
for (Entry<Key,Value> kv : scan) {
assertTrue(false);
}
Vertex v1 = graph.addVertex(id);
v1.setProperty(key, value);
Iterable<Element> elements = graph.getGlobals()
.getVertexKeyIndexWrapper().readElementsFromIndex(key, value);
assertEquals(0, count(elements));
}
@Test
public void testEdgeAutoIndex() throws Exception {
AccumuloGraph graph = (AccumuloGraph) GraphFactory.open(AccumuloGraphTestUtils
.generateGraphConfig("EdgeAutoIndex").setAutoIndex(true).getConfiguration());
String id1 = "A";
String id2 = "B";
String eid = "X";
String key = "name";
String value = "bananaman";
Vertex v1 = graph.addVertex(id1);
Vertex v2 = graph.addVertex(id2);
Edge e = graph.addEdge(eid, v1, v2, "edge");
e.setProperty(key, value);
Iterable<Element> elements = graph.getGlobals()
.getEdgeKeyIndexWrapper().readElementsFromIndex(key, value);
int count = 0;
for (Element element : elements) {
assertTrue(element instanceof Edge);
assertEquals(eid, element.getId());
assertEquals(value, element.getProperty(key));
count++;
}
assertEquals(1, count);
graph.removeVertex(v1);
elements = graph.getGlobals()
.getEdgeKeyIndexWrapper().readElementsFromIndex(key, value);
assertEquals(0, count(elements));
}
@Test
public void testEdgeNoAutoIndex() throws Exception {
AccumuloGraph graph = (AccumuloGraph) GraphFactory.open(AccumuloGraphTestUtils
.generateGraphConfig("EdgeNoAutoIndexTest").getConfiguration());
String id1 = "A";
String id2 = "B";
String eid = "X";
String key = "name";
String value = "bananaman";
Vertex v1 = graph.addVertex(id1);
Vertex v2 = graph.addVertex(id2);
Edge e = graph.addEdge(eid, v1, v2, "edge");
e.setProperty(key, value);
Iterable<Element> elements = graph.getGlobals()
.getEdgeKeyIndexWrapper().readElementsFromIndex(key, value);
assertEquals(0, count(elements));
}
@SuppressWarnings("unused")
private static int count(Iterable<?> it) {
int count = 0;
for (Object obj : it) {
count++;
}
return count;
}
}

View File

@@ -0,0 +1,39 @@
package edu.jhuapl.tinkerpop;
import org.apache.commons.configuration.Configuration;
import org.junit.Ignore;
import com.tinkerpop.blueprints.Graph;
import com.tinkerpop.blueprints.GraphFactory;
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration.InstanceType;
/**
* Run the test suite for a Distributed instance type.
* <p/>Note: This is disabled by default since we can't
* guarantee an Accumulo cluster setup in a test environment.
* To use this, set the constants below and remove
* the @Ignore annotation.
*/
@Ignore
public class DistributedInstanceTest extends AccumuloGraphTest {
// Connection constants.
private static final String ZOOKEEPERS = "zkhost-1,zkhost-2";
private static final String INSTANCE = "accumulo-instance";
private static final String USER = "user";
private static final String PASSWORD = "password";
@Override
public Graph generateGraph(String graphDirectoryName) {
Configuration cfg = new AccumuloGraphConfiguration()
.setInstanceType(InstanceType.Distributed)
.setZooKeeperHosts(ZOOKEEPERS)
.setInstanceName(INSTANCE)
.setUser(USER).setPassword(PASSWORD)
.setGraphName(graphDirectoryName)
.setCreate(true);
testGraphName.set(graphDirectoryName);
return GraphFactory.open(cfg);
}
}

View File

@@ -0,0 +1,108 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop;
import org.junit.Test;
import static org.junit.Assert.*;
import com.tinkerpop.blueprints.Edge;
import com.tinkerpop.blueprints.Element;
import com.tinkerpop.blueprints.Graph;
import com.tinkerpop.blueprints.GraphFactory;
import com.tinkerpop.blueprints.Vertex;
import edu.jhuapl.tinkerpop.cache.ElementCache;
public class ElementCacheTest {
@Test
public void testElementCacheSize() throws Exception {
AccumuloGraphConfiguration cfg = AccumuloGraphTestUtils
.generateGraphConfig("elementCacheSize");
Graph graph = GraphFactory.open(cfg.getConfiguration());
Vertex[] verts = new Vertex[10];
for (int i = 0; i < verts.length; i++) {
verts[i] = graph.addVertex(i);
}
Edge[] edges = new Edge[9];
for (int i = 0; i < edges.length; i++) {
edges[i] = graph.addEdge(null,
verts[0], verts[i+1], "edge");
}
sizeTests(verts);
sizeTests(edges);
graph.shutdown();
}
private void sizeTests(Element[] elts) {
ElementCache<Element> cache =
new ElementCache<Element>(3, 120000);
for (Element e : elts) {
cache.cache(e);
}
for (Element e : elts) {
cache.cache(e);
}
int total = 0;
for (Element e : elts) {
if (cache.retrieve(e.getId()) != null) {
total++;
}
}
assertTrue(total < elts.length);
cache.clear();
for (Element e : elts) {
assertNull(cache.retrieve(e.getId()));
}
}
@Test
public void testElementCacheTimeout() throws Exception {
AccumuloGraphConfiguration cfg = AccumuloGraphTestUtils
.generateGraphConfig("elementCacheTimeout");
Graph graph = GraphFactory.open(cfg.getConfiguration());
ElementCache<Element> cache =
new ElementCache<Element>(10, 1000);
Vertex v1 = graph.addVertex(1);
Vertex v2 = graph.addVertex(2);
assertNull(cache.retrieve(1));
assertNull(cache.retrieve(2));
cache.cache(v1);
assertNotNull(cache.retrieve(v1.getId()));
Thread.sleep(1500);
assertNull(cache.retrieve(v1.getId()));
Edge e = graph.addEdge(null, v1, v2, "label");
assertNull(cache.retrieve(e.getId()));
cache.cache(e);
assertNotNull(cache.retrieve(e.getId()));
Thread.sleep(1500);
assertNull(cache.retrieve(e.getId()));
graph.shutdown();
}
}

View File

@@ -0,0 +1,273 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop;
import static org.junit.Assert.*;
import org.junit.Test;
import com.google.common.collect.Sets;
import com.tinkerpop.blueprints.Element;
import com.tinkerpop.blueprints.Graph;
import com.tinkerpop.blueprints.GraphFactory;
/**
* Tests related to {@link Element}-based property
* loading and caching.
*/
public class ElementPropertyCachingTest {
private static final int TIMEOUT = 300000;
private static final String NON_CACHED = "noncached";
private static final String CACHED = "cached";
@Test
public void testCachingDisabled() {
AccumuloGraphConfiguration cfg =
AccumuloGraphTestUtils.generateGraphConfig("cachingDisabled");
assertTrue(cfg.getPropertyCacheTimeout(null) <= 0);
assertTrue(cfg.getPropertyCacheTimeout(NON_CACHED) <= 0);
assertTrue(cfg.getPropertyCacheTimeout(CACHED) <= 0);
Graph graph = open(cfg);
load(graph);
AccumuloVertex a = (AccumuloVertex) graph.getVertex("A");
AccumuloVertex b = (AccumuloVertex) graph.getVertex("B");
AccumuloVertex c = (AccumuloVertex) graph.getVertex("C");
assertEquals(null, a.getProperty(NON_CACHED));
assertEquals(true, b.getProperty(NON_CACHED));
assertEquals(null, c.getProperty(NON_CACHED));
assertEquals(null, a.getProperty(CACHED));
assertEquals(null, b.getProperty(CACHED));
assertEquals(true, c.getProperty(CACHED));
assertEquals(null, a.getPropertyCache().get(NON_CACHED));
assertEquals(null, b.getPropertyCache().get(NON_CACHED));
assertEquals(null, c.getPropertyCache().get(NON_CACHED));
assertEquals(null, a.getPropertyCache().get(CACHED));
assertEquals(null, b.getPropertyCache().get(CACHED));
assertEquals(null, c.getPropertyCache().get(CACHED));
assertEquals(Sets.newHashSet(), a.getPropertyCache().keySet());
assertEquals(Sets.newHashSet(), b.getPropertyCache().keySet());
assertEquals(Sets.newHashSet(), c.getPropertyCache().keySet());
a.removeProperty(NON_CACHED);
b.removeProperty(NON_CACHED);
c.removeProperty(NON_CACHED);
a.removeProperty(CACHED);
b.removeProperty(CACHED);
c.removeProperty(CACHED);
assertEquals(null, a.getProperty(NON_CACHED));
assertEquals(null, b.getProperty(NON_CACHED));
assertEquals(null, c.getProperty(NON_CACHED));
assertEquals(null, a.getProperty(CACHED));
assertEquals(null, b.getProperty(CACHED));
assertEquals(null, c.getProperty(CACHED));
assertEquals(null, a.getPropertyCache().get(NON_CACHED));
assertEquals(null, b.getPropertyCache().get(NON_CACHED));
assertEquals(null, c.getPropertyCache().get(NON_CACHED));
assertEquals(null, a.getPropertyCache().get(CACHED));
assertEquals(null, b.getPropertyCache().get(CACHED));
assertEquals(null, c.getPropertyCache().get(CACHED));
assertEquals(Sets.newHashSet(), a.getPropertyCache().keySet());
assertEquals(Sets.newHashSet(), b.getPropertyCache().keySet());
assertEquals(Sets.newHashSet(), c.getPropertyCache().keySet());
graph.shutdown();
}
@Test
public void testSpecificCaching() {
AccumuloGraphConfiguration cfg =
AccumuloGraphTestUtils.generateGraphConfig("getProperty");
cfg.setPropertyCacheTimeout(CACHED, TIMEOUT);
assertTrue(cfg.getPropertyCacheTimeout(null) <= 0);
assertTrue(cfg.getPropertyCacheTimeout(NON_CACHED) <= 0);
assertEquals(TIMEOUT, cfg.getPropertyCacheTimeout(CACHED));
Graph graph = open(cfg);
load(graph);
AccumuloVertex a = (AccumuloVertex) graph.getVertex("A");
AccumuloVertex b = (AccumuloVertex) graph.getVertex("B");
AccumuloVertex c = (AccumuloVertex) graph.getVertex("C");
assertEquals(null, a.getProperty(NON_CACHED));
assertEquals(true, b.getProperty(NON_CACHED));
assertEquals(null, c.getProperty(NON_CACHED));
assertEquals(null, a.getProperty(CACHED));
assertEquals(null, b.getProperty(CACHED));
assertEquals(true, c.getProperty(CACHED));
assertEquals(null, a.getPropertyCache().get(NON_CACHED));
assertEquals(null, b.getPropertyCache().get(NON_CACHED));
assertEquals(null, c.getPropertyCache().get(NON_CACHED));
assertEquals(null, a.getPropertyCache().get(CACHED));
assertEquals(null, b.getPropertyCache().get(CACHED));
assertEquals(true, c.getPropertyCache().get(CACHED));
assertEquals(Sets.newHashSet(), a.getPropertyCache().keySet());
assertEquals(Sets.newHashSet(), b.getPropertyCache().keySet());
assertEquals(Sets.newHashSet(CACHED), c.getPropertyCache().keySet());
a.removeProperty(NON_CACHED);
b.removeProperty(NON_CACHED);
c.removeProperty(NON_CACHED);
a.removeProperty(CACHED);
b.removeProperty(CACHED);
c.removeProperty(CACHED);
assertEquals(null, a.getProperty(NON_CACHED));
assertEquals(null, b.getProperty(NON_CACHED));
assertEquals(null, c.getProperty(NON_CACHED));
assertEquals(null, a.getProperty(CACHED));
assertEquals(null, b.getProperty(CACHED));
assertEquals(null, c.getProperty(CACHED));
assertEquals(null, a.getPropertyCache().get(NON_CACHED));
assertEquals(null, b.getPropertyCache().get(NON_CACHED));
assertEquals(null, c.getPropertyCache().get(NON_CACHED));
assertEquals(null, a.getPropertyCache().get(CACHED));
assertEquals(null, b.getPropertyCache().get(CACHED));
assertEquals(null, c.getPropertyCache().get(CACHED));
assertEquals(Sets.newHashSet(), a.getPropertyCache().keySet());
assertEquals(Sets.newHashSet(), b.getPropertyCache().keySet());
assertEquals(Sets.newHashSet(), c.getPropertyCache().keySet());
graph.shutdown();
}
@Test
public void testAllCaching() {
AccumuloGraphConfiguration cfg =
AccumuloGraphTestUtils.generateGraphConfig("setProperty");
cfg.setPropertyCacheTimeout(null, TIMEOUT);
cfg.setPropertyCacheTimeout(CACHED, TIMEOUT);
assertEquals(TIMEOUT, cfg.getPropertyCacheTimeout(null));
assertEquals(TIMEOUT, cfg.getPropertyCacheTimeout(NON_CACHED));
assertEquals(TIMEOUT, cfg.getPropertyCacheTimeout(CACHED));
Graph graph = open(cfg);
load(graph);
AccumuloVertex a = (AccumuloVertex) graph.getVertex("A");
AccumuloVertex b = (AccumuloVertex) graph.getVertex("B");
AccumuloVertex c = (AccumuloVertex) graph.getVertex("C");
assertEquals(null, a.getProperty(NON_CACHED));
assertEquals(true, b.getProperty(NON_CACHED));
assertEquals(null, c.getProperty(NON_CACHED));
assertEquals(null, a.getProperty(CACHED));
assertEquals(null, b.getProperty(CACHED));
assertEquals(true, c.getProperty(CACHED));
assertEquals(null, a.getPropertyCache().get(NON_CACHED));
assertEquals(true, b.getPropertyCache().get(NON_CACHED));
assertEquals(null, c.getPropertyCache().get(NON_CACHED));
assertEquals(null, a.getPropertyCache().get(CACHED));
assertEquals(null, b.getPropertyCache().get(CACHED));
assertEquals(true, c.getPropertyCache().get(CACHED));
assertEquals(Sets.newHashSet(), a.getPropertyCache().keySet());
assertEquals(Sets.newHashSet(NON_CACHED), b.getPropertyCache().keySet());
assertEquals(Sets.newHashSet(CACHED), c.getPropertyCache().keySet());
a.removeProperty(NON_CACHED);
b.removeProperty(NON_CACHED);
c.removeProperty(NON_CACHED);
a.removeProperty(CACHED);
b.removeProperty(CACHED);
c.removeProperty(CACHED);
assertEquals(null, a.getProperty(NON_CACHED));
assertEquals(null, b.getProperty(NON_CACHED));
assertEquals(null, c.getProperty(NON_CACHED));
assertEquals(null, a.getProperty(CACHED));
assertEquals(null, b.getProperty(CACHED));
assertEquals(null, c.getProperty(CACHED));
assertEquals(null, a.getPropertyCache().get(NON_CACHED));
assertEquals(null, b.getPropertyCache().get(NON_CACHED));
assertEquals(null, c.getPropertyCache().get(NON_CACHED));
assertEquals(null, a.getPropertyCache().get(CACHED));
assertEquals(null, b.getPropertyCache().get(CACHED));
assertEquals(null, c.getPropertyCache().get(CACHED));
assertEquals(Sets.newHashSet(), a.getPropertyCache().keySet());
assertEquals(Sets.newHashSet(), b.getPropertyCache().keySet());
assertEquals(Sets.newHashSet(), c.getPropertyCache().keySet());
graph.shutdown();
}
@Test
public void testPreloadAllProperties() {
AccumuloGraphConfiguration cfg =
AccumuloGraphTestUtils.generateGraphConfig("preloadAllProperties");
cfg.setPropertyCacheTimeout(null, TIMEOUT);
cfg.setPreloadAllProperties(true);
Graph graph = open(cfg);
AccumuloVertex v = (AccumuloVertex) graph.addVertex("V");
v.setProperty(NON_CACHED, true);
v.setProperty(CACHED, true);
v = (AccumuloVertex) graph.getVertex("V");
assertEquals(true, v.getPropertyInMemory(NON_CACHED));
assertEquals(true, v.getPropertyInMemory(CACHED));
graph.shutdown();
}
@Test
public void testPreloadSomeProperties() {
AccumuloGraphConfiguration cfg =
AccumuloGraphTestUtils.generateGraphConfig("preloadSomeProperties");
cfg.setPropertyCacheTimeout(null, TIMEOUT);
cfg.setPreloadedProperties(new String[]{CACHED});
Graph graph = open(cfg);
AccumuloVertex v = (AccumuloVertex) graph.addVertex("V");
v.setProperty(NON_CACHED, true);
v.setProperty(CACHED, true);
v = (AccumuloVertex) graph.getVertex("V");
assertEquals(null, v.getPropertyInMemory(NON_CACHED));
assertEquals(true, v.getPropertyInMemory(CACHED));
graph.shutdown();
}
private static Graph open(AccumuloGraphConfiguration cfg) {
return GraphFactory.open(cfg);
}
private static void load(Graph graph) {
graph.addVertex("A");
graph.addVertex("B").setProperty(NON_CACHED, true);
graph.addVertex("C").setProperty(CACHED, true);
}
}

View File

@@ -4,11 +4,16 @@ import com.tinkerpop.blueprints.Graph;
import com.tinkerpop.blueprints.GraphFactory;
public class ExtendedAccumuloGraphTest extends AccumuloGraphTest {
@Override
public Graph generateGraph(String graphDirectoryName) {
AccumuloGraphConfiguration cfg = AccumuloGraphTestUtils.generateGraphConfig(graphDirectoryName);
cfg.setLruMaxCapacity(20).setPreloadedProperties(new String[] {"name"}).setPreloadedEdgeLabels(new String[] {"knows"}).setPropertyCacheTimeout("name",100000);
cfg.setEdgeCacheParams(20, 30000)
.setPreloadedProperties(new String[] {"name"})
.setPreloadedEdgeLabels(new String[] {"knows"})
.setPropertyCacheTimeout("name", 100000);
testGraphName.set(graphDirectoryName);
return GraphFactory.open(cfg.getConfiguration());
}
}

View File

@@ -0,0 +1,107 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop;
import static org.junit.Assert.*;
import org.junit.Test;
import com.tinkerpop.blueprints.Graph;
import com.tinkerpop.blueprints.GraphFactory;
import com.tinkerpop.blueprints.Vertex;
/**
* Tests related to implementation-specific elements.
*/
public class ExtendedElementTest {
private Graph makeGraph(AccumuloGraphConfiguration cfg) {
return GraphFactory.open(cfg.getConfiguration());
}
@Test
public void testExistenceChecks() throws Exception {
AccumuloGraphConfiguration cfg =
AccumuloGraphTestUtils.generateGraphConfig("yesExistenceChecks");
Graph graph = makeGraph(cfg);
String id;
id = "doubleAdd";
assertNotNull(graph.addVertex(id));
try {
graph.addVertex(id);
fail();
} catch (Exception e) { }
Vertex vAdd = graph.getVertex(id);
assertNotNull(vAdd);
graph.removeVertex(vAdd);
assertNull(graph.getVertex(id));
id = "doubleRemove";
Vertex vRemove = graph.addVertex(id);
assertNotNull(vRemove);
graph.removeVertex(vRemove);
try {
graph.removeVertex(vRemove);
fail();
} catch (Exception e) { }
assertNull(graph.getVertex(id));
id = "notExist";
assertNull(graph.getVertex(id));
graph.shutdown();
}
@Test
public void testSkipExistenceChecks() throws Exception {
AccumuloGraphConfiguration cfg =
AccumuloGraphTestUtils.generateGraphConfig("skipExistenceChecks");
cfg.setSkipExistenceChecks(true);
Graph graph = makeGraph(cfg);
String id;
id = "doubleAdd";
assertNotNull(graph.addVertex(id));
assertNotNull(graph.addVertex(id));
Vertex vAdd = graph.getVertex(id);
assertNotNull(vAdd);
graph.removeVertex(vAdd);
assertNotNull(graph.getVertex(id));
id = "doubleRemove";
Vertex vRemove = graph.addVertex(id);
assertNotNull(vRemove);
graph.removeVertex(vRemove);
assertNotNull(graph.getVertex(id));
// MDL 24 Dec 2014: removeVertex still does checks.
//graph.removeVertex(vRemove);
//assertNotNull(graph.getVertex(id));
id = "notExist";
assertNotNull(graph.getVertex(id));
graph.shutdown();
}
}

View File

@@ -0,0 +1,23 @@
package edu.jhuapl.tinkerpop;
import org.apache.commons.configuration.Configuration;
import com.tinkerpop.blueprints.Graph;
import com.tinkerpop.blueprints.GraphFactory;
import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration.InstanceType;
/**
* Run the test suite for a Mock instance type.
*/
public class MockInstanceTest extends AccumuloGraphTest {
@Override
public Graph generateGraph(String graphDirectoryName) {
Configuration cfg = new AccumuloGraphConfiguration()
.setInstanceType(InstanceType.Mock)
.setGraphName(graphDirectoryName);
testGraphName.set(graphDirectoryName);
return GraphFactory.open(cfg);
}
}

View File

@@ -0,0 +1,87 @@
/* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package edu.jhuapl.tinkerpop;
import static org.junit.Assert.*;
import org.junit.Test;
import edu.jhuapl.tinkerpop.cache.PropertyCache;
/**
* Test the {@link PropertyCache} object.
*/
public class PropertyCacheTest {
@Test
public void testUncachedProperty() throws Exception {
AccumuloGraphConfiguration cfg =
AccumuloGraphTestUtils.generateGraphConfig("uncached");
PropertyCache cache = new PropertyCache(cfg);
cache.put("K1", "V1");
cache.put("K2", "V2");
cache.put("K3", "V3");
assertNull(cache.get("K1"));
assertNull(cache.get("K2"));
assertNull(cache.get("K3"));
}
@Test
public void testDefaultCachedProperty() throws Exception {
AccumuloGraphConfiguration cfg =
AccumuloGraphTestUtils.generateGraphConfig("defaultCached");
cfg.setPropertyCacheTimeout(null, 1000);
PropertyCache cache = new PropertyCache(cfg);
cache.put("K1", "V1");
assertEquals("V1", cache.get("K1"));
Thread.sleep(1500);
assertNull(cache.get("K1"));
cache.put("K2", "V2");
assertEquals("V2", cache.get("K2"));
cache.remove("K2");
assertNull(cache.get("K2"));
cache.put("K3", "V3");
cache.clear();
assertNull(cache.get("K3"));
}
@Test
public void testSpecificCachedProperty() throws Exception {
AccumuloGraphConfiguration cfg =
AccumuloGraphTestUtils.generateGraphConfig("specificCached");
cfg.setPropertyCacheTimeout(null, 1000);
cfg.setPropertyCacheTimeout("long", 2000);
cfg.setPropertyCacheTimeout("longer", 3000);
PropertyCache cache = new PropertyCache(cfg);
cache.put("default", "V1");
cache.put("long", "V2");
cache.put("longer", "V3");
assertEquals("V1", cache.get("default"));
assertEquals("V2", cache.get("long"));
assertEquals("V3", cache.get("longer"));
Thread.sleep(1500);
assertNull(cache.get("default"));
Thread.sleep(1000);
assertNull(cache.get("long"));
Thread.sleep(1000);
assertNull(cache.get("longer"));
}
}

View File

@@ -136,35 +136,50 @@ public class InputFormatsTest {
@Test
public void testVertexInputMap() throws Exception {
final String INSTANCE_NAME = "_mapreduce_instance";
final String TEST_TABLE_1 = "_mapreduce_table_1";
final String TEST_TABLE_NAME = "_mapreduce_table_vertexInputMap";
if (!System.getProperty("os.name").startsWith("Windows")) {
Graph g = GraphFactory.open(new AccumuloGraphConfiguration().setInstanceName(INSTANCE_NAME).setUser("root").setPassword("".getBytes())
.setGraphName(TEST_TABLE_1).setInstanceType(InstanceType.Mock).setCreate(true).getConfiguration());
Graph g = GraphFactory.open(new AccumuloGraphConfiguration().setInstanceName(INSTANCE_NAME)
.setUser("root").setPassword("".getBytes())
.setGraphName(TEST_TABLE_NAME).setInstanceType(InstanceType.Mock)
.setCreate(true).getConfiguration());
for (int i = 0; i < 100; i++) {
g.addVertex(i + "");
}
assertEquals(0, MRTester.main(new String[] {"root", "", TEST_TABLE_1, INSTANCE_NAME, "false"}));
assertEquals(0, MRTester.main(new String[]{"root", "",
TEST_TABLE_NAME, INSTANCE_NAME, "false"}));
assertNull(e1);
assertNull(e2);
g.shutdown();
}
}
@Test
public void testEdgeInputMap() throws Exception {
final String INSTANCE_NAME = "_mapreduce_instance";
final String TEST_TABLE_1 = "_mapreduce_table_1";
final String TEST_TABLE_NAME = "_mapreduce_table_edgeInputMap";
if (!System.getProperty("os.name").startsWith("Windows")) {
Graph g = GraphFactory.open(new AccumuloGraphConfiguration().setInstanceName(INSTANCE_NAME).setUser("root").setPassword("".getBytes())
.setGraphName(TEST_TABLE_1).setInstanceType(InstanceType.Mock).autoFlush(true).setCreate(true).getConfiguration());
for (int i = 0; i < 100; i++) {
g.addEdge(null, g.addVertex(i + ""), g.addVertex(i + "a"), "knows");
Graph g = GraphFactory.open(new AccumuloGraphConfiguration().setInstanceName(INSTANCE_NAME)
.setUser("root").setPassword("".getBytes())
.setGraphName(TEST_TABLE_NAME).setInstanceType(InstanceType.Mock)
.setAutoFlush(true).setCreate(true).getConfiguration());
for (int i = 0; i < 100; i++) {
Vertex v1 = g.addVertex(i+"");
Vertex v2 = g.addVertex(i+"a");
g.addEdge(null, v1, v2, "knows");
}
assertEquals(0, MRTester.main(new String[] {"root", "", TEST_TABLE_1, INSTANCE_NAME, "true"}));
assertEquals(0, MRTester.main(new String[]{"root", "",
TEST_TABLE_NAME, INSTANCE_NAME, "true"}));
assertNull(e1);
assertNull(e2);
g.shutdown();
}
}

93
table-structure.md Normal file
View File

@@ -0,0 +1,93 @@
AccumuloGraph Table Schema
==========================
AccumuloGraph uses a number of backing tables for data storage.
This file documents the structure and organization of these
tables. The tables and records are structured in such a way as
to implement the Blueprints operations in terms of efficient Accumulo
operations, i.e., prefix searches, contiguous scans, and
batch operations.
For our purposes, Accumulo entries consist of four fields:
row (R), column family (CF), column qualifier (CQ), and value (V).
For more information on Accumulo's schema and how to
access records efficiently, see the
[Accumulo documentation](https://accumulo.apache.org/1.5/accumulo_user_manual.html).
The tables used by AccumuloGraph are prefixed by the configured graph
name and can be classed as element/property tables, and
index tables. Their structure is discussed below.
Elements and properties
-----------------------
Vertex and edge information, along with their properties, are stored
in the *graphname*\_vertex and *graphname*\_edge tables respectively.
First, an entry declares the existence of a vertex in the vertex table.
| R | CF | CQ | V |
|---|----|----|---|
| *vertex_id* | `_LABEL_` | `_EXISTS_`| *[empty]* |
A similar entry declares the existence of an edge in the edge table.
| R | CF | CQ | V |
|---|----|----|---|
| *edge_id* | `_LABEL_` | *in_vertex_id*`_DELIM_`*out_vertex_id* | *edge_label* |
When adding an edge, additional entries are stored in the
vertex table for each endpoint of the edge. These facilitate the
`Vertex.getEdge` and `Vertex.getVertex` operations.
| R | CF | CQ | V |
|---|----|----|---|
| *in_vertex_id* | `_IN_EDGE_` | *out_vertex_id*`_DELIM_`*edge_id* | *edge_label* |
| *out_vertex_id* | `_OUT_EDGE_` | *in_vertex_id*`_DELIM_`*edge_id* | *edge_label* |
Finally, vertex and edge properties are stored in their respective
tables. Entry formats are the same for both vertices and edges.
Note that property values are serialized such that their type
can be deduced when deserializing.
| R | CF | CQ | V |
|---|----|----|---|
| *element_id* | *property_key* | *[empty]* | *property_value* |
Indexes
-------
Several tables store index-related information,
including index value tables that store index
property keys and values, and index metadata tables
that store information about what indexes exist
and what properties are indexed.
For `KeyIndexableGraph`, index value tables
include *graphname*\_vertex\_key\_index
and *graphname*\_edge\_key\_index for vertex
and edge properties, respectively.
For `IndexableGraph`, index value tables are
named *graphname*\_index\_*indexname*,
where *indexname* is the index name.
The entry formats in all these tables are the same:
| R | CF | CQ | V |
|---|----|----|---|
| *property_value* | *property_key* | *element_id* | *[empty]* |
Property values are serialized in the same way as above.
In addition to the index value tables, an index metadata table,
*graphname*\_index\_metadata, stores index information. For
`KeyIndexableGraph`, records in this table enumerate the property keys
that are indexed.
| R | CF | CQ | V |
|---|----|----|---|
| *property_key* | `_INDEX_KEY_` | *element_class* | *[empty]* |
For `IndexableGraph`, records enumerate the existing indexes.
| R | CF | CQ | V |
|---|----|----|---|
| *index_name* | `_INDEX_NAME_` | *element_class* | *[empty]* |