Neo4J

Neo4j – Uber H3 – Geospatial

February 19, 2019 by dave fauth Leave a Comment

We are going to take a slight detour with regards to the healthcare blog series and talk about Uber H3. H3 is a hexagonal hierarchical geospatial indexing system. It comes with an API for indexing coordinates into a global grid. The grid is fully global and you can choose your resolution. The advantages and disadvantages of the hexagonal grid system are discussed here and here . Uber open-sourced the H3 indexing system and it comes with a set of java bindings that we will use with Neo4j.

Since version 3.4, Neo4j has a native geospatial datatype. Neo4j uses the WGS-84 and WGS-84 3D coordinate reference system. Within Neo4j, we can index these point properties and query using our distance function or you query within a bounding box. Max DeMarzi blogged about this as did Neo4j. At GraphConnect 2018, Will Lyon and Craig Taverner had a session on Neo4j and Going Spatial. These are all great resources for Neo4j and geo-spatial search.

Why Uber H3

Uber H3 adds some new features that can be extended through Neo4j procedures. Specifically, we want to be able to query based on a polygon, query based on a polygon with holes, and even along a line. We will use the data from our healthcare demo dataset and show how we can use H3 hexagon addresses to help our queries.

H3 Procedure

For our first procedure, we will pass in the latitude, longitude and resolution and receive a hexAddress back.

The H3 interface also allows us to query via a polygon. With Neo4j, we can query using a bounding box like so:

In H3, the polygon search looks like this:

This returns in 17ms against 4.4 million locations.

If we combine this with our healthcare data, we can find providers with a certain taxonomy.

We can make this polygon as complicated as we need. This example is for a county polygon:

CALL com.dfauth.h3.polygonSearch([{lon:"-77.302457",lat:"38.504683"},{lon:"-77.310334",lat:"38.493926"},{lon:"-77.322622",lat:"38.467131"},{lon:"-77.32544",lat:"38.44885"},{lon:"-77.319036",lat:"38.417803"},{lon:"-77.310719",lat:"38.397669"},{lon:"-77.312201",lat:"38.390958"},{lon:"-77.314848",lat:"38.389579"},{lon:"-77.317288",lat:"38.383576"},{lon:"-77.296077",lat:"38.369797"},{lon:"-77.288145",lat:"38.359477"},{lon:"-77.28835",lat:"38.351286"},{lon:"-77.286202",lat:"38.347025"},{lon:"-77.286202",lat:"38.347024"},{lon:"-77.321403",lat:"38.345226"},{lon:"-77.339268",lat:"38.302723"},{lon:"-77.345728",lat:"38.26139"},{lon:"-77.326692",lat:"38.245136"},{lon:"-77.370301",lat:"38.246576"},{lon:"-77.39085",lat:"38.245589"},{lon:"-77.420148",lat:"38.257986"},{lon:"-77.447126",lat:"38.284614"},{lon:"-77.455692",lat:"38.301341"},{lon:"-77.467053",lat:"38.31866"},{lon:"-77.475137",lat:"38.32096"},{lon:"-77.478996",lat:"38.316693"},{lon:"-77.498754",lat:"38.32543"},{lon:"-77.506782",lat:"38.325925"},{lon:"-77.527185",lat:"38.320655"},{lon:"-77.526243",lat:"38.309531"},{lon:"-77.530289",lat:"38.309172"},{lon:"-77.54546",lat:"38.325081"},{lon:"-77.584673",lat:"38.346806"},{lon:"-77.594796",lat:"38.336022"},{lon:"-77.618727",lat:"38.367835"},{lon:"-77.634835",lat:"38.409713"},{lon:"-77.628433",lat:"38.452075"},{lon:"-77.634157",lat:"38.464508"},{lon:"-77.568349",lat:"38.520177"},{lon:"-77.530914",lat:"38.555929"},{lon:"-77.481488",lat:"38.592432"},{lon:"-77.463949",lat:"38.578686"},{lon:"-77.448683",lat:"38.580792"},{lon:"-77.395824",lat:"38.545827"},{lon:"-77.370142",lat:"38.519865"},{lon:"-77.334902",lat:"38.514569"},{lon:"-77.308138",lat:"38.499699"},{lon:"-77.310528",lat:"38.505289"},{lon:"-77.302457",lat:"38.504683"}],[{}]) yield nodes return nodes

In these examples, there are no “donut holes”. If I need a polygon donut hole to exclude an area, I can pass it in as the second parameter.

CALL com.dfauth.h3.polygonSearch([{lon:"-77.302457",lat:"38.504683"},{lon:"-77.310334",lat:"38.493926"},{lon:"-77.322622",lat:"38.467131"},{lon:"-77.32544",lat:"38.44885"},{lon:"-77.319036",lat:"38.417803"},{lon:"-77.310719",lat:"38.397669"},{lon:"-77.312201",lat:"38.390958"},{lon:"-77.314848",lat:"38.389579"},{lon:"-77.317288",lat:"38.383576"},{lon:"-77.296077",lat:"38.369797"},{lon:"-77.288145",lat:"38.359477"},{lon:"-77.28835",lat:"38.351286"},{lon:"-77.286202",lat:"38.347025"},{lon:"-77.286202",lat:"38.347024"},{lon:"-77.321403",lat:"38.345226"},{lon:"-77.339268",lat:"38.302723"},{lon:"-77.345728",lat:"38.26139"},{lon:"-77.326692",lat:"38.245136"},{lon:"-77.370301",lat:"38.246576"},{lon:"-77.39085",lat:"38.245589"},{lon:"-77.420148",lat:"38.257986"},{lon:"-77.447126",lat:"38.284614"},{lon:"-77.455692",lat:"38.301341"},{lon:"-77.467053",lat:"38.31866"},{lon:"-77.475137",lat:"38.32096"},{lon:"-77.478996",lat:"38.316693"},{lon:"-77.498754",lat:"38.32543"},{lon:"-77.506782",lat:"38.325925"},{lon:"-77.527185",lat:"38.320655"},{lon:"-77.526243",lat:"38.309531"},{lon:"-77.530289",lat:"38.309172"},{lon:"-77.54546",lat:"38.325081"},{lon:"-77.584673",lat:"38.346806"},{lon:"-77.594796",lat:"38.336022"},{lon:"-77.618727",lat:"38.367835"},{lon:"-77.634835",lat:"38.409713"},{lon:"-77.628433",lat:"38.452075"},{lon:"-77.634157",lat:"38.464508"},{lon:"-77.568349",lat:"38.520177"},{lon:"-77.530914",lat:"38.555929"},{lon:"-77.481488",lat:"38.592432"},{lon:"-77.463949",lat:"38.578686"},{lon:"-77.448683",lat:"38.580792"},{lon:"-77.395824",lat:"38.545827"},{lon:"-77.370142",lat:"38.519865"},{lon:"-77.334902",lat:"38.514569"},{lon:"-77.308138",lat:"38.499699"},{lon:"-77.310528",lat:"38.505289"},{lon:"-77.302457",lat:"38.504683"}],[{lon:'-77.530886',lat:'38.3609911'},{lon:'-77.524492',lat:'38.3411698'},{lon:'-77.524492',lat:'38.277433'},{lon:'-77.5731773',lat:'38.2774607'},{lon:'-77.594635',lat:'38.2771873'}]) yield nodes return nodes

In the 3.3.0 release of the Java bindings, H3 included the ability to return all hex addresses along a line between two points. One could combine this with a service like Google Directions to find all locations along a route. Imagine finding doctors along a route or find all events that occurred along a route.
Here is an example that returns providers who have billing addresses along a line:

CALL com.dfauth.h3.lineBetweenLocations(38.418582, -77.385268,38.500603, -77.444288) yield nodes 
unwind nodes as locNode 
match (locNode)<-[:HAS_BILLING_ADDRESS_AT]-(p:Provider)-[:HAS_SPECIALTY]->(t:TaxonomyCode) 
return distinct locNode.Address1 + ' ' + locNode.CityName +', ' + locNode.StateName as locationAddress, locNode.latitude as latitude, locNode.longitude as longitude, coalesce(p.BusinessName,p.FirstName + ' ' + p.LastName) as practiceName, p.NPI as NPI
order by locationAddress;

Pretty neat, right? As always, the source code as always is available on github.

Neo4j – Leveraging a graph for healthcare search

February 18, 2019 by dave fauth 1 Comment

Graph-based search is intelligent: You can ask much more precise and useful questions and get back the most relevant and meaningful information, whereas traditional keyword-based search delivers results that are more random, diluted and low-quality.

With graph-based search, you can easily query all of your connected data in real time, then focus on the answers provided and launch new real-time searches prompted by the insights you’ve discovered.

Recently, I’ve been developing proof-of-concept (POC) demonstrations showing the power of graph-based search with healthcare data. In this series of blog posts, we will implement a data model that powers graph-based search solutions for the healthcare community.

Our first scenario will be to use Neo4j for some simple search and find a provider based on location and/or specialty. As my colleague Max DeMarzi says, “The optimal model is heavily dependent on the queries you want to ask”. Queries that we want to ask are things like: For the State of Virginia, can I find a provider by my location? Where is the nearest pharmacy? Can we break that down by County or by County and Specialty? The queries drive our model. The model that we will use for this scenario and one we will build on later looks like this:

What we see is that there is a Provider who has a Specialty and multiple locations (practice, billing and alternate practice). Postal codes are in a county and postal codes are also a certain distance in miles from each other. We also have loaded the DocGraph Referral/Patient Sharing Data set.

Data Sources
The US Government’s Centers for Medicare and Medicaid Services (CMS) has a whole treasure trove of data resources that we can use to load into Neo4j and use for our POC.

Our first dataset is the NPPES Data Dissemination File. This file contains information on healthcare providers in the US. The key field is the NPI which is a unique 10-digit identification number issued to health care providers in the United States by the CMS.

Our second dataset is a summary spreadsheet showing the number of patients in managed services by county. I’ll use this file to generate patients per county and distribute them across the US.

Our third dataset is the Physician and Other Supplier Public Use File (Physician and Other Supplier PUF). This file provides information on services and procedures provided to Medicare beneficiaries by physicians and other healthcare professionals. The Physician and Other Supplier PUF contains information on utilization, payment (allowed amount and Medicare payment), and submitted charges organized by National Provider Identifier (NPI), Healthcare Common Procedure Coding System (HCPCS) code, and place of service. I’ll use this file for data validation and for building out provider specialties.

Our fourth dataset is the Public Provider Enrollment Data. The Public Provider Enrollment data for Medicare fee-for-service includes providers who are actively approved to bill Medicare or have completed the 855O at the time the data was pulled from the Provider Enrollment and Chain Ownership System (PECOS). These files are populated from PECOS and contain basic enrollment and provider information, reassignment of benefits information and practice location city, state and zip.

Finally, we have a dataset that maps US Counties to US Zip Codes and a dataset that maps the distance between zip codes.

Data Downloads:
You can download the data files from:

NPPES Data Dissemination File
NUCC Taxonomy
Counties to Zip Code / FIPS
Zip Code Tabulation Area
Zip Code Distance

Data Loading

In the next post, I will document the data loading procedures that I wrote for this data.

Additional Information – Public Training (March 2019)

For the first time, datapalooza is hosting pre-conference classes on open data.

Here is the link to signup, you do not need to attend datapalooza to attend the classes.

https://www.academyhealth.org/page/2019-hdp-pre-conference-events

We are offering training on the following open datasets at this years datapalooza:

Introduction to NPI and NPPES
Introduction to NDC data
Referral/Patient Sharing Data Tutorial (will cover all versions of the docgraph dataset)
Open Payments data

Modeling events in Neo4j to look for patterns

February 18, 2019 by dave fauth Leave a Comment

Recently, some of the prospects that I work with have wanted to understand event data and felt like a graph database would be the best approach. Being new to graphs, they aren’t always sure of the best modeling approach.

There are some resources available that talk about modeling events. For example, Neo4j’s own Mark Needham published a blog post showing modeling TV shows among other events.. Snowplow published a recent blog post that describes a similar data model.

One thing that we always try to stress is to build the model to effectively answer the questions you ask of the graph. In this prospect’s case, they were collecting item scans that occurred within a warehouse. Each item had a series of scans as it moved through the warehouse. Based on the individual item or even a type of item, could they discover common patterns or deviations from the normal pattern? Another use case for this would be to track how users traverse a website or how students complete an on-line course.

An initial approach might be to create a relationship between each item and the scanner. The model is shown below:

An (:Item)-[:HAS_TAG]->(:ItemTag)-[:SCANNED]->(:Scanner) is the Cypher pattern for this model.

In this model, it is easy to understand how many times an :ItemTag was scanned. Over time, a Scanner could easily become a supernode where we have millions of :SCANNED relationships attached to the node. Furthermore, if we want to find common patterns, we have to look at all :SCANNED relationships, order them by a property and then do comparisons. The amount of work increases as the amount of :SCANNED relationships increase. This is similar to the AIRPORT example in Max’s blogpost.

So what can we do? Let’s change the model to be a series of SCAN events that form a series of events from the initial scan to the final scan. For the relationship types, instead of using :NEXT or :PREVIOUS, we will name the relationship types by the scanner id. Let’s look at that model:

In this model, an :Item has an :ItemDay. This allows us to manage and search all tags by item and by day without creating supernodes. Between each :SCAN node, we use a specific relationship type that indicates the scanner. This allows us to easily find patterns within the data. Neo4j allows for over 64k custom relationship types, so you should be good with this approach.

If you were modeling a TV show, you may want to track when a NEW episode was show versus a REPEAT episode. You could model that as:
(:Episode)-[:NEW]->(:Episode)-[:REPEAT]->(:Episode)-[:REPEAT]->(:Episode)

For an item, we can find the most common patterns using the below Cypher code:

match (r:Tag)
with r
match path=(r)-[re*..50]->(:Scan)
with r.TagID as TagID,  [r IN re | TYPE(r)] as paths
order by length(path) desc 
with collect(paths) as allPaths, TagID
return count(TagID) as tagCount, head(allPaths)
order by tagCount desc;

The use of the variable length relationship allows us to easily find those patterns no matter the length. If we want to find an anomaly such as where :SCANNER_101 wasn’t the first scan, we could do something like:

match path=(a1:Tag)-[r2]->(a:Scan)-[r1:SCANNER_101]->(b:Scan)
return count(path);

With the ability to add native date/time properties to a node, we can use Cypher to calculate durations between event scans. For example, this snippet shows the minimum, maximum and average duration between scans for each type of scan.

match (a:Scan)-[r1]->(b:Scan)
with r1, duration.between(a.scanDateTime,b.scanDateTime) as theduration
return type(r1), max(theduration.minutes), min(theduration.minutes), avg(theduration.minutes)
order by type(r1) asc, avg(theduration.minutes) desc;

When modeling data in Neo4j, think of how you want to traverse the graph to get the answers that you need. The answers will inevitably determine the shape of the graph and the model that you choose.

Resources:

Accessing Hive/Impala from Neo4j

November 16, 2015 by dave fauth Leave a Comment

Quite frequently, I get asked about how you could import data from Hadoop and bring it into Neo4j. More often than not, the request is about importing from Impala or Hive. Today, Neo4j isn’t able to directly access an HDFS file system so we have to use a different approach.

For this example, we will use a Neo4j Unmanaged Extension. Unmanaged extensions provide us with the ability to hook into the native Java API and expand the capabilities of the Neo4j server. The unmanaged extension is reached via a REST API. With the extension, we can connect to Impala/Hive, query the database, return the data and then manipulate the data based on what we need.

Let’s create a simple example of an Java unmanaged extension which exposes a Neo4j REST endpoint that connects to an Hive instance running on another machine, query the database and create nodes.

What is Cloudera Impala
A good overview of Impala is in this presentation on SlideShare. Impala provides interactive SQL, nearly ANSI-92 standard SQL queries and provides a JDBC connector allowing external tools to access Impala. Cloudera Impala is the industry’s leading massively parallel processing (MPP) SQL query engine that runs natively in Apache Hadoop. The Apache-licensed, open source Impala project combines modern, scalable parallel database technology with the power of Hadoop, enabling users to directly query data stored in HDFS and Apache HBase without requiring data movement or transformation.

What is Cloudera Hive
From Cloudera, “Hive enables analysis of large data sets using a language very similar to standard ANSI SQL. This means anyone who can write SQL queries can access data stored on the Hadoop cluster. This discussion introduces the functionality of Hive, as well as its various applications for data analysis and data warehousing.”

Installing a Hadoop Cluster
For this demonstration, we will use the Cloudera QuickStart VM running on VirtualBox. You can download the VM and run it in VMWare, KVM and VirtualBox. I chose VirtualBox. The VM gives you access to a working Impala and Hive instance without manually installing each of the components.

Connecting to Impala/Hive
Cloudera provides a download page for the JDBC driver. Once you download the jdbc driver, you will want to unzip it and copy the jar files from the Cloudera_ImpalaJDBC4_2.5.5.1007 directory into your plugins directory.

Hortonworks
For Hortonworks, you can download a Sandbox for VirtualBox or VMWare. Once installed, you will need to set the IP address for the Hortonworks machine. The Hortonworks sandbox has the same sample data set in Hive.

Unmanaged Extension
Our unmanaged extension (code below), queries against the default database in Hive and creates an equipment node with two properties for each record in the sample_07 table. The query is hardcoded but could easily be passed in as a parameter with the POST request.

Deploying
1. Build the Jar
2. Copy target/neo4j-hadoop.jar to the plugins/ directory of your Neo4j server.
3. Copy the Cloudera JDBC jar files to your plugins directory.
4. Configure Neo4j by adding a line to conf/neo4j-server.properties:

org.neo4j.server.thirdparty_jaxrs_classes=com.neo4j.hadoop.example=/v1

5. Check that it is installed correctly over HTTP:

:GET /v1/service/helloworld

6. Query the database:

:POST /v1/service/equipment

7. Open the Neo4j browser and query the Equipment nodes:

MATCH (e:Equipment) return count(e);

Results in Neo4j

Summary
Cloudera’s JDBC driver combined with Neo4J’s Unmanaged Extension provides an easy and fast way to connect to a Hadoop instance and populate a Neo4J instance. While this example doesn’t show relationships, you can easily modify the example to create the relationships.

Full code for the project is on Github.

Source Code for the example is below:

Neo4j – New Neo4j-import

December 17, 2014 by dave fauth Leave a Comment

Neo4j has recently announced the 2.2 Milestone 2 release. Among the exciting features is the improved and fully integrated “Superfast Batch Loader”. This utility (unsurprisingly) called neo4j-import, now supports large scale non-transactional initial loads (of 10M to 10B+ elements) with sustained throughputs around 1M records (node or relationship or property) per second. Neo4j-import is available from the command line on both Windows and Unix.

In this post, I’ll walk through how to set up the data files, some command line options and then document the performance for importing a medium size data set.

Data set

The data set that we will use is Medicare Provider Utilization and Payment Data.

The Physician and Other Supplier PUF contains information on utilization, payment (allowed amount and Medicare payment), and submitted charges organized by National Provider Identifier (NPI), Healthcare Common Procedure Coding System (HCPCS) code, and place of service. This PUF is based on information from CMS’s National Claims History Standard Analytic Files. The data in the Physician and Other Supplier PUF covers calendar year 2012 and contains 100% final-action physician/supplier Part B non-institutional line items for the Medicare fee-for-service population.

For the data model, I created doctor nodes, address nodes, procedure nodes and procedure detail nodes.

A (doctor)-[:LOCATED_AT]-(Address)
A (doctor)-[:PERFORMED]-(procedure)
A (procedure) -[:CONTAINS]-(procedure_details)

The model is shown below:

Import Tool Notes

The new neo4j-import tool has the following capabilities:

Fields default to be comma separated, but a different delimiter can be specified.
All files must use the same delimiter.
Multiple data sources can be used for both nodes and relationships.
A data source can optionally be provided using multiple files.
A header which provides information on the data fields must be on the first row of each data source.
Fields without corresponding information in the header will not be read.
UTF-8 encoding is used.

File Layouts

The header file must have an :ID and can have multiple properties and a :LABEL value. The :ID is a unique value that is used during node creation but more importantly is used during relationships creation. The :ID value is used later as the :START_ID or :END_ID value which are used to create the relationships. :ID values can be integers or strings as long as they are unique.

The :LABEL value is used to create the Label in the Property Graph. A label is a named graph construct that is used to group nodes into sets; all nodes labeled with the same label belongs to the same set.

For example:

:ID	Name	:LABEL	NPI
2	ARDALAN ENKESHAFI	Internal Medicine	1003000126
7	JENNIFER VELOTTA	Obstetrics/Gynecology	1003000423
8	JACLYN EVANS	Clinical Psychologist	1003000449

In this case, the :ID is the unique number, and the :LABEL column sets the doctor type. Name and NPI are property values added to the node.

The relationships file header contains a :START_ID, :END_ID and can optionally include properties and :TYPE. The :START_ID and the :END_ID link back to previously :ID values from the nodes files. The :TYPE is used to set the relationship type. All relationships in Neo4j have a relationship type.

:START_ID	:END_ID	:TYPE
2	19627026	LOCATED_AT
3	19397586	LOCATED_AT
4	19446780	LOCATED_AT
5	19266134	LOCATED_AT

Separate Header Files
It can be convenient to put the header in a separate file. This makes it easier to edit the header, as you avoid having to open a huge data file to just change the header. To use a separate header file in the command line, see the example below:

neo4j-import --into path_to_target_directory 
--nodes "docNodes-header.csv docNodes.csv" 
--nodes "addressNodes-header.csv addressNodes.csv" 
--relationships "docLocationRelationships-header.csv docLocationRelationships.csv"

Command Line Usage

Under Unix/Linux/OSX, the command is named neo4j-import. Under Windows, the command is named Neo4jImport.bat.

Depending on the installation type, the tool is either available globally, or used by executing ./bin/neo4j-import or bin\Neo4jImport.bat from inside the installation directory.

Additional details can be found online.

Loading Medicare Data using neo4j-import

From the command line, we run the neo4j-import command. The –into option is used to indicate where to store the new database. The directory must not contain an existing database.

The –nodes option indicate the header and nodes files. Multiple –nodes can be used.
The –relationships option indicates the header and relationship files. Multiple –relationships can be used.

./bin/neo4j-import --into /Users/davidfauth/tempDB 
--nodes /medicareData/docNodes.csv 
--nodes /medicareData/addressNodes.csv 
--nodes /medicareData/procedureDetailNodes.csv 
--nodes /medicareData/procedureNodes.csv  
--relationships /medicareData/docLocationRelationships.csv 
--relationships /medicareData/procedureToProcedureRels.csv 
--relationships /medicareData/docProcedureRelationships.csv 
--delimiter TAB

Results

Running the load script on my 16GB Macbook Pro with an SSD, I was able to load 20.7M nodes, 23.4M relationships and 35.6M properties in 2 minutes and 5 seconds.

Conclusion

The new neo4j-import tool provides significant performance improvements and ease-of-use for the initial data loading of a Neo4j database. Please download the latest milestone release and give the neo4j-import tool a try.

We are eager to hear your feedback. Please post it to the Neo4j Google Group, or send us a direct email at feedback@neotechnology.com.

Hadoop, Impala and Neo4J

March 12, 2014 by dave fauth Leave a Comment

Back in December, I wrote about some ways of moving data from Hadoop into Neo4J using Pig, Py2Neo and Neo4J. Overall, it was successful although maybe not at the scale I would have liked. So this is really attempt number two at using Hadoop technology to populate a Neo4J instance.

In this post, I’ll use a combination of Neo4J’s batchInserter plus JDBC to query against Cloudera’s Impala which in turn queries against files on HDFS. We’ll use a list of 694,221 organizations that I had from previous docGraph work.

Installing a Hadoop Cluster
For this demonstration, we will use the Cloudera QuickStart VM running on VirtualBox. You can download the VM and run it in VMWare, KVM and VirtualBox. I chose VirtualBox. The VM gives you access to a working Impala instance without manually installing each of the components.

Sample Data
As mentioned above, the dataset is a list of 694,221 organizations that I had from previous docGraph work. Each organization is a single name that we will use to create organization nodes in Neo4J. An example is shown below:

BMH, INC
BNNC INC
BODYTEST
BOSH INC
BOST INC
BRAY INC

Creating a table
After starting the Cloudera VM, I logged into Cloudera Manager and made sure Impala was starting. Once Impala was up and running, I used the Impala Query Editor to create the table.

DROP TABLE if exists organizations;
CREATE EXTERNAL TABLE organizations
(c_orgname	string)
row format delimited fields terminated by '|';

This creates an organizations table with a single c_orgname column.

Populating the Table
Populating the Impala table is really easy. After copying the source text file to the VM and dropping it into an HDFS directory, I ran the following command in the Impala query interface:

load data inpath '/user/hive/dsf/sample/organizationlist1.csv' into table organizations;

I had three organization list files. When I was done, I did a “Select * from organizations” resulting in a count of 694,221 records.

Querying the table
Cloudera provides a page on the JDBC driver. This page enables you to download the JDBC driver as well as points you to a GitHub page with a sample Java program for querying against Impala.

Populating Neo4J with BatchInserter and JDBC
While there are a multitude of ways of populating Neo4J, in this case I modified an existing program that I had written around the BatchInserter to populate Neo4J. The code can be browsed on Github. The java code creates an empty Neo4J data directory. It then connects to Impala through JDBC and runs a query to get all of the organizations. The code then loops over the result set, determines if a node with the name exists and creates a node with the organizational name if the node did not previously exist.

Running the program over the 694,221 records in Impala, the Neo4J database was created in approximately 20 seconds. Once that is done, I copied the database files into my Neo4J instance and restarted.

Once inside of Neo4J, I can query on the data or visualize the information as shown below.

Summary
Impala’s JDBC driver combined with Neo4J’s BatchInserter provides an easy and fast way to populate a Neo4J instance. While this example doesn’t show relationships, you can easily modify the BatchInserter to create the relationships.

Blogroll

Archives

Contact Me