Upgrading to Apache TinkerPop 3.8.x in Amazon Neptune 1.5.0.0 brings cleaner, more predictable Gremlin query behavior and a major expansion of Data Flow Engine (DFE) coverage. It also introduces semantic changes that require careful query review. The 3.7.x line was almost entirely additive, with more than 20 new Gremlin steps for working with strings, collections, and dates. TinkerPop 3.8.0 instead refines and, in some cases, changes the semantics of the Gremlin language itself. Some changes present as compile-time errors with a clear fix, but others are more subtle and surface only at query runtime. This version asks for some review up front. The upside is a cleaner, more predictable Gremlin with new in-query type-conversion and type-testing steps for mixed-type data. It also brings a major expansion of DFE coverage for Gremlin in Neptune.

In this post, you will learn what changed, which query patterns are affected, and how to prepare for the upgrade. This post intentionally focuses on Neptune and doesn’t cover the full TinkerPop change list. For the authoritative, complete description of every change, see the Apache TinkerPop 3.8.0 Upgrade Documentation. The examples here use the 1.0 version of the air routes dataset.

Before you upgrade

To use Gremlin with Neptune 1.5.0.0, you must use the TinkerPop 3.8.x release line. At the time of writing, the latest version is 3.8.2. Don’t use earlier release lines, such as 3.7.x, because they aren’t compatible. Drivers might connect, but you will likely encounter runtime errors.

You can send Gremlin to Neptune in several ways. You can use a Gremlin driver over WebSockets, the AWS SDK (which uses HTTP), or the Graph Notebook. Regardless of the method you use, one of the most important steps in this upgrade is to carefully review the Gremlin queries in your workload. Check whether any of them match the patterns described in this post. Then determine whether you need to make changes so that your queries continue to return the same results as in prior versions. You aren’t only looking for errors when you upgrade, because the same queries might succeed with different output. Therefore, compare the query results on 1.5.0.0 with the results on the version of Neptune you were using to verify that they match. Also test queries that match these modified patterns across a wide range of data that triggers alternate traversal paths.

Some of the changes to Gremlin require relatively advanced knowledge of the language terminology to fully grasp the nature of the semantic differences between 3.7.x and 3.8.x. Diving into this terminology is beyond the scope of this post. Consider refreshing your memory on the definition of a Traverser and the role of Barrier steps in a Gremlin query. A good foundation of Gremlin concepts helps you follow this post and prepares you for a smooth upgrade.

Gremlin changes to watch for

The following sections list the Gremlin changes to look for in your code.

Date and time values and dateDiff()

If you use Java, the default Gremlin date type changes from java.util.Date to java.time.OffsetDateTime. The date steps you already use, specifically asDate(), dateAdd(), dateDiff(), and the datetime() helper, now return OffsetDateTime, whose string form is ISO-8601. Depending on your usage, this change might present as a clear compilation error, but it’s worth reviewing any queries that currently work with java.util.Date.

An equally significant change, whatever programming language you use, is that dateDiff() now returns milliseconds, where it previously returned seconds. This is the change most likely to quietly affect application code at runtime and requires review of all dateDiff() step usage. If you parse date output, or if you divide a dateDiff() result expecting seconds, you need to adjust your code. A value that used to read -86400 (seconds in a day) now reads -86400000.

// 1.4.8.0/3.7.x - seconds are returned gremlin> g.inject("2023-08-02T00:00:00Z").asDate(). ......1> dateDiff(constant("2023-08-03T00:00:00Z").asDate()) ==>-86400 // 1.5.0.0/3.8.x - milliseconds are returned gremlin> g.inject("2023-08-02T00:00:00Z").asDate(). ......1> dateDiff(constant("2023-08-03T00:00:00Z").asDate()) ==>-86400000 // 1.5.0.0/3.8.x math() and asNumber() can recover seconds if you // need them gremlin> g.inject("2023-08-02T00:00:00Z").asDate(). ......1> dateDiff(constant("2023-08-03T00:00:00Z").asDate()). ......2> math("_/1000").asNumber(GType.LONG) ==>-86400

For more information, see New Default DateTime Type and dateDiff() Step Returns Milliseconds.

Numeric type auto-promotion

Two related changes affect the type of numbers your queries produce. Floating-point literals now default to double rather than float in the Gremlin language. Arithmetic operations now auto-promote to a larger type on overflow instead of silently wrapping around.

For Neptune workloads with mixed-type numeric properties, the practical impact is that the type of a returned value can differ from prior versions. Review anything sensitive to the exact numeric type of a result: serialization assumptions, equality checks, and downstream math.

// 1.4.8.0/3.7.x - values wrap around within their type on overflow gremlin> g.V().has('airport','code', within('IAD','DCA','JFK')). ......1> values('runways'). ......2> inject(2147483647i).fold().sum(local) ==>-2147483638 // 1.5.0.0/3.8.x - the result auto-promotes to the next wider type // instead of wrapping gremlin> g.V().has('airport','code', within('IAD','DCA','JFK')). ......1> values('runways'). ......2> inject(2147483647i).fold().sum(local) ==>2147483658

Only overflow triggers promotion. Arithmetic that doesn’t overflow returns the common type of its operands, exactly as in prior versions.

For more information, see Float Defaults to Double and Auto-promotion of Numbers.

Removed and renamed steps

TinkerPop removed or renamed a few steps, either to close long-standing sources of misuse or to make usage more consistent. There are three patterns to look for. First, the none() step was renamed to discard() to make room for the new none(P) step that complements all(P) and any(P). You can generally fix this with a find and replace.

Second, the form of has() that takes a String key and Traversal as arguments was removed entirely because of its misunderstood semantics. If you currently use has(String,Traversal) in your code, it might not be doing what you think. It’s natural to assume that using has() this way compares the value of the key you provide to the value returned by the Traversal argument, but that is incorrect. It actually takes the value of the key and injects that value into the Traversal, essentially testing whether it returns any result at all. To get the semantics that you want, you usually replace this has() step form with the where() step.

For more information, see the Using ‘where’ and ‘by’ to filter results section of Practical Gremlin.

// 1.4.8.0/3.7.x - Proper but non-idiomatic usage of // has(String, Traversal) gremlin> g.V().has('runways', __.is(P.gt(6))).values('desc') ==>Dallas/Fort Worth International Airport ==>Chicago O'Hare International Airport // 1.5.0.0/3.8.x - do not wrap the predicate in a Traversal; // this form is valid with any version of TinkerPop or Neptune gremlin> g.V().has('runways', P.gt(6)).values('desc') ==>Dallas/Fort Worth International Airport ==>Chicago O'Hare International Airport // 1.4.8.0/3.7.x - Common misuse where you want to find airports // with the same number of runways as some vertex. Here // select('iad').values('runways') is just always true gremlin> g.V().has('code', 'IAD').as('iad'). ......1> out(). ......2> has('runways', select('iad').values('runways')). ......3> count() ==>158 // incidentally, a test of the count of outgoing vertices // demonstrates the ineffectiveness of the above use of // has(String,Traversal) gremlin> g.V().has('code', 'IAD').as('iad').out().count() ==>158 // 1.5.0.0/3.8.x - Prefer where() step to express this // query, which was true in 3.7.x and earlier versions as well gremlin> g.V().has('code', 'IAD').as('iad'). ......1> out().as('o'). ......2> where('o', P.eq('iad')).by('runways').count() ==>20

The third and final step-removal pattern to scan for involves store() and aggregate(Scope, String). TinkerPop removed both steps. At a high level, the TinkerPop community wanted a unified control mechanism for lazy versus eager evaluation. This mechanism gives providers more flexibility to apply optimizations and gives users the freedom to constrain flow when necessary. TinkerPop also had a specific reason to remove the aggregate(Scope, String) overload. It conflated the core semantics of Scope as a way to specify a traverser-local mode for a step with flow control for a step. The aggregate() step was the only step to do the latter, controlling whether the aggregation occurred lazily for Scope.local or eagerly for Scope.global.

If you currently use aggregate(Scope, String) with the Scope.global argument, remove the Scope.global argument from your code. Most users of the global form likely called it without Scope.global already as a shorthand.

// 1.4.8.0/3.7.x - using aggregate(global) gremlin> g.V().has('runways',gt(5)).aggregate(Scope.global,'v'). ......1> cap('v').count(local) ==>6 // 1.5.0.0/3.8.x - remove the Scope.global argument gremlin> g.V().has('runways',gt(5)).aggregate('v').cap('v'). ......1> count(local) ==>6

If you currently use store() or aggregate(Scope, String) with the Scope.local argument, there is more to consider. Neptune never supported the lazy flow semantics that TinkerPop offered with these two steps. If you required those semantics, you likely wrapped those steps in a local() step to trigger them. Determine whether your Gremlin query actually required the lazy flow semantics that were implied by those steps. Often, switching from lazy to eager makes little difference to the results of the query. If you did not require that flow control, change that step to use aggregate(String) and use the standard eager semantics of the step.

On the other hand, if your algorithm did require some measure of flow control, use the local() step. If you implemented flow control correctly, you are likely already doing this, because you would have found that store() and aggregate(local) differed semantically on Neptune compared to TinkerPop. You likely used patterns similar to those in the following example, which demonstrates the difference between TinkerPop and Neptune semantics.

// TinkerPop 3.7.x - store() and aggregate(local) accumulate one // airport's runways one at a time gremlin> g.V().has('airport','code',within('ATL','AUS','DFW')). ......1> order().by('code'). ......2> store('x').by('runways'). ......3> select('x') ==>[5] ==>[5,2] ==>[5,2,7] gremlin> g.V().has('airport','code',within('ATL','AUS','DFW')). ......1> order().by('code'). ......2> aggregate(local,'x').by('runways'). ......3> select('x') ==>[5] ==>[5,2] ==>[5,2,7] // TinkerPop 3.8.x - wrap aggregate() in local() to accumulate one // airport's runways one at a time gremlin> g.V().has('airport','code',within('ATL','AUS','DFW')). ......1> order().by('code'). ......2> local(aggregate('x').by('runways').select('x')) ==>[5] ==>[5,2] ==>[5,2,7] // Neptune (all versions) - wrap aggregate() in local() with the // addition of by(unfold().fold()) to simulate lazy/local accumulation gremlin> g.V().has('airport','code',within('ATL','AUS','DFW')). ......1> order().by('code'). ......2> local(aggregate('x').by('runways'). ......3>. select('x').by(unfold().fold())) ==>[5] ==>[5,2] ==>[5,2,7]

For more information, see Removal of aggregate() with Scope and store().

repeat() changes

Because repeat() is one of the most commonly used steps in Gremlin, you likely have some queries to review in this context. Looping traversals see several changes, and this area requires careful examination and testing. There are three key areas to consider.

First, a mechanical change prevents cap() and inject() from being used inside repeat() and raises a runtime error if they are present. If you currently use these steps in repeat(), examine their behavior carefully. Your query might rely on inconsistencies in the semantics of those steps and likely needed a rewrite in any case. Using union() or constant() inside repeat() for per-loop injection might provide the semantics your query requires. Consider those options as a replacement.

Second, if the child traversal that you pass to repeat() contains limit(), skip(), or range(), note that these steps now apply to each iteration rather than as a global counter across loops. Move them outside repeat() if you need them to apply across all loops.

// 1.5.0.0/3.8.x - takes a limit of 2 at each iteration gremlin> g.V().has('airport','code','AUS'). ......1> repeat(out('route').order().by('code').limit(2)).times(2). ......2> values('code') ==>ATL ==>AUS

Achieving the semantics shown in the preceding example often required a complicated Gremlin pattern, described in the first edition of the book Practical Gremlin in the Limiting the results at each depth section. If you use such a pattern, replace it with this more readable form.

Third, Gremlin now consistently treats the traversal that you pass to repeat() as a global child. This change affects ordering and how barriers and side effects inside repeat() behave.

// 1.4.8.0/3.7.x - order is local to the first traverser where you can // see that results are grouped by ERM and then ULP gremlin> g.V().has('airport','code', within('ERM','ULP')). ......1> repeat(out('route').simplePath().order().by('code')). ......2> times(2). ......3> path().by('code') ==>[ERM,JCB,CFC] ==>[ERM,JCB,FLN] ==>[ERM,PFB,GRU] ==>[ERM,PFB,POA] ==>[ERM,PFB,VCP] ==>[ULP,CTL,BNE] ==>[ULP,WNR,BVI] ==>[ULP,CTL,RMA] ==>[ULP,CTL,WTB] // 1.5.0.0/3.8.x - order is globally consistent where the traversers // from the final iteration are ordered first then by the traversers // from previous iterations gremlin> g.V().has('airport','code', within('ERM','ULP')). ......1> repeat(out('route').simplePath().order().by('code')). ......2> times(2). ......3> path().by('code') ==>[ULP,CTL,BNE] ==>[ULP,WNR,BVI] ==>[ERM,JCB,CFC] ==>[ERM,JCB,FLN] ==>[ERM,PFB,GRU] ==>[ERM,PFB,POA] ==>[ULP,CTL,RMA] ==>[ERM,PFB,VCP] ==>[ULP,CTL,WTB] // 1.5.0.0/3.8.x - wrapping the repeat in local() step can simulate // the 3.7.x semantics gremlin> g.V().has('airport','code', within('ERM','ULP')). ......1> local(repeat(out('route').simplePath(). ......2> order().by('code')).times(2)). ......3> path().by('code') ==>[ERM,JCB,CFC] ==>[ERM,JCB,FLN] ==>[ERM,PFB,GRU] ==>[ERM,PFB,POA] ==>[ERM,PFB,VCP] ==>[ULP,CTL,BNE] ==>[ULP,WNR,BVI] ==>[ULP,CTL,RMA] ==>[ULP,CTL,WTB]

For more information, see repeat() Step Global Children Semantics Change, Modified limit() skip() range() Semantics in repeat(), and Prevented using cap(), inject() inside repeat().

choose() changes

With the choose() step, you can write if-then-else logic in Gremlin. In 3.8.x, several changes make it work more consistently. Examine your use of choose() carefully, because the following changes can significantly modify query results after you upgrade. First, choose() now processes only the first matching option, rather than every matching option.

// 1.4.8.0/3.7.x - IAD, DCA and JFK have runway values of 3, 4, 4 and // therefore fall into both option ranges gremlin> g.V().has('airport','code', within('IAD','DCA','JFK')). ......1> choose(__.values("runways")). ......2> option(P.between(3, 5), __.constant("has runways")). ......3> option(P.between(0, 5), __.constant("also has runways")) ==>has runways ==>also has runways ==>has runways ==>also has runways ==>has runways ==>also has runways // 1.5.0.0/3.8.x - while the runway values still satisfy both option // ranges, the second option is ignored. gremlin> g.V().has('airport','code', within('IAD','DCA','JFK')). ......1> choose(__.values("runways")). ......2> option(P.between(3, 5), __.constant("has runways")). ......3> option(P.between(0, 5), __.constant("also has runways")) ==>has runways ==>has runways ==>has runways

If you need the 3.7.x semantics for choose(), use branch() instead.

// 1.4.8.0/3.7.x and 1.5.0.0/3.8.x semantically prefer branch() for a // case where you want to match on multiple options gremlin> g.V().has('airport','code', within('IAD','DCA','JFK')). ......1> branch(__.values("runways")). ......2> option(P.between(3, 5), __.constant("has runways")). ......3> option(P.between(0, 5), __.constant("also has runways")) ==>has runways ==>also has runways ==>has runways ==>also has runways ==>has runways ==>also has runways

The next change to choose() is in how it handles traversers that don’t match any option or whose choice traversal is unproductive. They are now passed through instead of producing an error or being filtered out respectively.

// 1.4.8.0/3.7.x - v[0] is not an airport vertex and does not have // runways and therefore produces an error since the choice // traversal is unproductive (i.e. __.values("runways") does not // produce a value for v[0]) gremlin> g.V('0', '7', '10', '12'). ......1> choose(__.values("runways")). ......2> option(P.between(3, 5), __.constant("has runways")). ......3> option(P.between(0, 5), __.constant("also has runways")) The provided traverser does not map to a value.... // 1.5.0.0/3.8.x - does not produce an error and will pass the // non-airport traverser through as a result gremlin> g.V('0', '7', '10', '12'). ......1> choose(__.values("runways")). ......2> option(P.between(3, 5), __.constant("has runways")). ......3> option(P.between(0, 5), __.constant("also has runways")) ==>v[0] ==>has runways ==>has runways ==>has runways // 1.4.8.0/3.7.x - 2 airports do not match 3 or 5 runways so they are // filtered gremlin> g.V().has('airport','code', within('IAD','DCA','JFK')). ......1> choose(__.values("runways")). ......2> option(3, __.constant("has 3 runways")). ......3> option(5, __.constant("has 5 runways")) ==>has 3 runways // 1.5.0.0/3.8.x - the 2 airports that do not match stated options // pass through choose unchanged gremlin> g.V().has('airport','code', within('IAD','DCA','JFK')). ......1> choose(__.values("runways")). ......2> option(3, __.constant("has 3 runways")). ......3> option(5, __.constant("has 5 runways")) ==>has 3 runways ==>v[10] ==>v[12]

You can capture either of these cases with Pick.none for no matches, and with a new Pick.unproductive token you can use to explicitly handle traversers whose choice traversal produced no value.

As a final semantic change to consider, TinkerPop removed the choose().option(Traversal, v) overload to better constrain choose() to its primary capability of testing values. Rewrite those cases with union() and prepended filters.

// 1.4.8.0/3.7.x - allowed a traversal to spawn from the match option gremlin> g.V().has('airport','code', within('IAD','DCA','JFK')). ......1> choose(identity()). ......2> option(outE().count().is(P.gt(100)), ......3>. valueMap('code','runways')). ......4> option(Pick.none, constant("none")) ==>none ==>[code:[IAD],runways:[4]] ==>[code:[JFK],runways:[4]] // 1.5.0.0/3.8.x - the same traversal now produces an error, but can // be rewritten more idiomatically with union gremlin> g.V().has('airport','code', within('IAD','DCA','JFK')). ......1> choose(identity()). ......2> option(outE().count().is(P.gt(100)), ......3> constant("big degree")). ......4> option(Pick.none, constant("small degree")) Traversal is not allowed as a Pick token for choose().option() gremlin> g.V().has('airport','code', within('IAD','DCA','JFK')). ......1> union(where(outE().count().is(P.gt(100))). ......2> valueMap('code','runways'), ......3> __.not(where(outE().count().is(P.gt(100)))). ......4> constant("none")) ==>[code:[IAD],runways:[4]] ==>[code:[JFK],runways:[4]] ==>none

For more information, see choose() Semantics.

by() modulator restrictions

valueMap(), propertyMap(), groupCount(), dedup(), sack(), sample(), and aggregate() now throw an error if more than one by() modulator is supplied. Previously, the extra modulators were silently ignored or unexpectedly applied in round-robin order. Any query that (perhaps accidentally) chained multiple by() modulators on these steps now fails. Reduce them to a single by() when you upgrade. For more information, see By Modulation Semantics.

Collection output

TinkerPop 3.8.x makes Gremlin more consistent in the results it returns and the way steps behave. One of the longest-held inconsistencies involved range(Scope, int), limit(Scope, int), and tail(Scope, int) when used with Scope.local. In this usage under 3.7.x, these steps returned a list if a collection had zero items or more than one item. If there was only a single item, the step unwrapped that item from the list. This behavior was originally offered as a convenience for working with one object, but its consistency was always questioned. In 3.8.x, you can always expect a list from these steps.

// 1.4.8.0/3.7.x - note that DCA is not a list object - it has been // unwrapped gremlin> g.V().has('airport','code', within('IAD','DCA','JFK')). ......1> values('code').fold().limit(local,0) ==>[] gremlin> g.V().has('airport','code', within('IAD','DCA','JFK')). ......1> values('code').order().fold().limit(local,1) ==>DCA gremlin> g.V().has('airport','code', within('IAD','DCA','JFK')). ......1> values('code').order().fold().limit(local,2) ==>[DCA,IAD] // 1.5.0.0/3.8.x - results are consistently a list object gremlin> g.V().has('airport','code', within('IAD','DCA','JFK')). ......1> values('code').order().fold().limit(local,0) ==>[] gremlin> g.V().has('airport','code', within('IAD','DCA','JFK')). ......1> values('code').order().fold().limit(local,1) ==>[DCA] gremlin> g.V().has('airport','code', within('IAD','DCA','JFK')). ......1> values('code').order().fold().limit(local,2) ==>[DCA,IAD]

If you use step variations of (local, 1) and want to retain the single object unwrapped from the list, add unfold() after the step to get the same semantics as 3.7.x.

For more information, see Consistent Output for range(), limit(), tail().

Other behavior changes to review

Scan your code for the following smaller changes:

  • split("") on an empty separator now splits a string into its characters.
  • asString() no longer accepts null input (it throws an exception), for consistency with asDate(), asBool(), and asNumber().
  • In a group() value traversal, an empty result from a barrier (such as order()) is no longer dropped. For more information, see Group Value Traversal Semantics.
  • Mid-traversal mergeV() and mergeE() now pass the created or matched Element to the onCreate and onMatch options. For more information, see Element Promoted for mergeV() / mergeE().
  • Comparisons between incomparable types, or against NaN, now resolve to false instead of raising an error.

New capabilities you can adopt

Alongside the semantic changes, TinkerPop 3.8.x adds new steps that are useful for Neptune. Consider the new predicate P.typeOf(GType), which filters traversers by runtime type. This predicate helps you detect property data that is stored inconsistently across your graph, and it also supports other use cases. For example, when paths contain a mix of vertices and edges, it can be hard to separate the edges from the vertices:

gremlin> g.V().has('airport','code', 'IAD'). ......1> outE().inV().path().limit(5) ==>[v[10],e[6144][10-route->1053],v[1053]] ==>[v[10],e[6145][10-route->1059],v[1059]] ==>[v[10],e[6146][10-route->1060],v[1060]] ==>[v[10],e[6147][10-route->1061],v[1061]] ==>[v[10],e[6148][10-route->1062],v[1062]]

The traditional workaround has been to find a property key unique to the element type and use that key as a filter. If you use a pattern like that, you can switch to the more readable typeOf approach.

gremlin> g.V().has('airport','code', 'IAD'). ......1> outE().inV().path().limit(5). ......2> map(unfold().is(P.typeOf(GType.VERTEX)).fold()) ==>[v[10],v[1053]] ==>[v[10],v[1059]] ==>[v[10],v[1060]] ==>[v[10],v[1061]] ==>[v[10],v[1062]]

TinkerPop 3.8.x also introduces asNumber() and asBool(). The former parses or casts strings and numbers to a target numeric type (GType), and the latter parses strings and numbers into Boolean values. Together with the string, collection, and date manipulation steps introduced in 3.7.x, Gremlin now has a full library of type-normalization options. You can now perform these operations in queries rather than in application logic.

For more information, see Type Predicate, Number Conversion Step, and Boolean Conversion Step.

Considering barrier steps

In some cases, Neptune was not completely aligned with TinkerPop and the TinkerPop community chose to change its semantics to match Neptune. As a result, some changes that TinkerPop lists as breaking in its upgrade documentation don’t affect existing Neptune users. The key behavior to consider is described in the section on Add barrier to most SideEffect steps.

In these cases, aggregate(String), group(String), groupCount(String), tree(String), and subgraph(String) now behave as barrier steps rather than side effects, which matches their counterparts that don’t take a named side-effect string. As barrier steps, they now collect results eagerly, and this behavior is unchanged for Neptune in 1.5.0.0.

// A manufactured example of lazy accumulation semantics in // TinkerPop 3.7.x which Neptune has never supported gremlin> g.V().has('airport','code', within('IAD','DCA','JFK')). ......1> groupCount("x").select("x") ==>{v[7]=1} ==>{v[7]=1,v[10]=1} ==>{v[7]=1,v[10]=1,v[12]=1} // 1.5.0.0 and earlier versions of Neptune have the same behavior they // always have gremlin> g.V().has('airport','code', within('IAD','DCA','JFK')). ......1> groupCount("x").select("x") ==>{v[12]=1, v[7]=1, v[10]=1} ==>{v[12]=1, v[7]=1, v[10]=1} ==>{v[12]=1, v[7]=1, v[10]=1}

DFE and step coverage in 1.5.0.0

One of the most significant Neptune improvements in this release has nothing to do with the language specification. With 1.5.0.0, the Neptune DFE query engine significantly expands its Gremlin support, and DFE support for Gremlin is no longer experimental. DFE natively supports most Gremlin steps, which makes a wide variety of traversal patterns available on this engine for the first time. For example, if you use DFE, you can now use common steps like choose() and the wide range of string, date, and collection manipulation steps, such as conjoin(), dateDiff(), and product(). These steps now fully optimize on the DFE engine.

In addition to supporting new steps, DFE widens query patterns of previously supported steps. The following code snippet shows a few examples of expanded support.

// 1.5.0.0 - use of dedup(), limit() and as() inside of repeat() are // all supported gremlin> g.withSideEffect('Neptune#useDFE', true).V(). ......1> has('code','AGR'). ......2> repeat(out().dedup()).until(has('code','CCU')). ......3> path().by('code') ==>[AGR,BOM,CCU] gremlin> g.withSideEffect('Neptune#useDFE', true). ......1> V().has('code','AGR'). ......2> repeat(out().limit(5)).times(2). ......3> has('code','LHR').path().by('code') ==>[AGR,BOM,LHR] gremlin> g.withSideEffect('Neptune#useDFE', true).V(). ......1> has('code','AGR').as('a'). ......2> repeat(out().dedup().as('b')).until(has('code','CCU')). ......3> select('a','b').by('code') ==>[a:AGR,b:CCU]

DFE also improves support for and() and or() patterns, particularly with nesting.

gremlin> g.withSideEffect("Neptune#useDFE", true).V(). ......1> hasLabel('airport'). ......2> or(__.in('contains').hasLabel('country'). ......3> values('desc').is('United Kingdom'), ......4> __.out('route').count().is(gt(150))). ......5> limit(5). ......6> values('desc') ==>Hartsfield - Jackson Atlanta International Airport ==>Dallas/Fort Worth International Airport ==>Fort Lauderdale/Hollywood International Airport ==>San Francisco International Airport ==>Las Vegas Mc Carran gremlin> g.withSideEffect("Neptune#useDFE", true).V(). ......1> hasLabel('airport'). ......2> and(or(has('region','US-TX'), ......3>. has('code','LHR'), has('runways',gt(6))), ......4> or(out().count().is(gt(160)), has('code','AUS'))). ......5> project('code','region','count'). ......6> by('code').by('region').by(out().count()) ==>[code:AUS,count:98,region:US-TX] ==>[code:DFW,count:253,region:US-TX] ==>[code:LHR,count:221,region:GB-ENG] ==>[code:IAH,count:199,region:US-TX] ==>[code:ORD,count:265,region:US-IL]

In general, more queries of greater step complexity translate fully to DFE with 1.5.0.0. For example, the following Gremlin query finds the five highest-elevation airports and shapes them into a projected record. This common query pattern did not translate fully to DFE before 1.5.0.0, but DFE now has the step and pattern coverage to optimize it.

gremlin> g.withSideEffect("Neptune#useDFE", true).V(). ......1> order().by('elev', desc).limit(5). ......2> project('code','elev','city','country'). ......3> by('code').by('elev').by('city'). ......4> by(in('contains').hasLabel('country').values('desc')) ==>[city:Daocheng,code:DCY,elev:14472,country:China] ==>[city:Bangda,code:BPX,elev:14219,country:China] ==>[city:Kangding,code:KGT,elev:14042,country:China] ==>[city:Shiquanhe,code:NGQ,elev:14022,country:China] ==>[city:La Paz / El Alto,code:LPB,elev:13355,country:Bolivia]

With wider DFE support in 1.5.0.0, consider re-testing Gremlin queries that previously failed to translate fully on earlier versions. If they do now, you may find that they yield better query performance.

Conclusion

Apache TinkerPop 3.8.x, available in Amazon Neptune 1.5.0.0, is a maturing of the Gremlin language rather than a collection of new steps. TinkerPop replaces less intuitive semantics with behavior that is more predictable and easier to reason about. That maturity is worth adopting, and the DFE expansion means more of your Gremlin now runs on the more advanced Neptune engine than ever before.

Because some of these changes alter results rather than raising errors, the upgrade deserves a deliberate review:

  1. Clone and compare. Run your representative workload against a 1.5.0.0 test cluster and compare the results, not only the errors. Validate the results across a wide range of data and traversal paths.
  2. Start with the high-frequency patterns. Collection return consistency for range()-like steps, repeat()-related patterns, and store() and aggregate(local) rewrites cover the majority of real-world impact.
  3. Adopt the new steps where they help. P.typeOf(), asNumber(), and asBool() are a good fit for mixed-type graph data and can replace fragile application-side casting.

As always, read the TinkerPop 3.8.0 upgrade documentation and the CHANGELOG for the full list of changes. With your queries reviewed and your drivers updated, upgrading to Neptune 1.5.0.0 gives you a cleaner, more predictable Gremlin and a substantially broader native DFE footprint. For more information about the other features of 1.5.0.0, please read the Engine Release Notes.


About the author

Stephen Mallette

Stephen Mallette

Stephen is a member of the Amazon Neptune team at AWS. He has developed graph database and graph processing technology for many years and is a long-time contributor to the Apache TinkerPop project, the home of the Gremlin graph query language.