Skip to main content

Advanced Collection Data Types

For an interactive Jupyter notebook experience: Launch in Binder#

Last updated: June 22, 2021

The goal of this tutorial is to highlight the power of working with collection data types (CDTs) in Aerospike. It covers the following topics:

  1. Setting contexts (CTXs) to apply operations to nested Maps and Lists.
  2. Showing the return type options provided by CDT get/read operations.
  3. Highlighting how policies shape application transactions.

This Jupyter Notebook requires the Aerospike Database running locally with Java kernel and Aerospike Java Client. To create a Docker container that satisfies the requirements and holds a copy of these notebooks, visit the Aerospike Notebooks Repo.

Prerequisites#

This Notebook builds on the material in the following notebooks:

  1. Working with Lists
  2. Working with Maps
  3. Introduction to Transactions

It uses examples based on those from Modeling Using Lists and Working with Maps. If any of the following is confusing, please refer to a relevant notebook.

Notebook Setup#

Import Jupyter Java Integration#

Make it easier to work with Java in Jupyter.

import io.github.spencerpark.ijava.IJava;import io.github.spencerpark.jupyter.kernel.magic.common.Shell;
IJava.getKernelInstance().getMagics().registerMagics(Shell.class);

Start Aerospike#

Ensure Aerospike Database is running locally.

%sh asd

Download the Aerospike Java Client#

Ask Maven to download and install the project object model (POM) of the Aerospike Java Client.

%%loadFromPOM<dependencies>  <dependency>    <groupId>com.aerospike</groupId>    <artifactId>aerospike-client</artifactId>    <version>5.0.0</version>  </dependency></dependencies>

Start the Aerospike Java Client and Connect#

Create an instance of the Aerospike Java Client, and connect to the demo cluster.

The default cluster location for the Docker container is localhost port 3000. If your cluster is not running on your local machine, modify localhost and 3000 to the values for your Aerospike cluster.

import com.aerospike.client.AerospikeClient;
AerospikeClient client = new AerospikeClient("localhost", 3000);System.out.println("Initialized the client and connected to the cluster.");

Output:

Initialized the client and connected to the cluster.

Create CDT Data, Put into Aerospike, and Print It#

import com.aerospike.client.Key;import com.aerospike.client.Bin;import com.aerospike.client.policy.ClientPolicy;import com.aerospike.client.Record;import com.aerospike.client.Operation;import com.aerospike.client.Value;import com.aerospike.client.cdt.ListOperation;import com.aerospike.client.cdt.ListPolicy;import com.aerospike.client.cdt.ListOrder;import com.aerospike.client.cdt.ListWriteFlags;import com.aerospike.client.cdt.MapOperation;import com.aerospike.client.cdt.MapPolicy;import com.aerospike.client.cdt.MapOrder;import com.aerospike.client.cdt.MapWriteFlags;

import java.util.ArrayList;import java.util.Arrays;import java.util.HashMap;import java.util.List;import java.util.Map;

// Create whale migration list of tuples. 
ArrayList<Value> whaleMigration0 = new ArrayList<Value>();whaleMigration0.add(Value.get(1420));whaleMigration0.add(Value.get("beluga whale"));whaleMigration0.add(Value.get("Beaufort Sea"));whaleMigration0.add(Value.get("Bering Sea"));
ArrayList<Value> whaleMigration1 = new ArrayList<Value>();whaleMigration1.add(Value.get(13988));whaleMigration1.add(Value.get("gray whale"));whaleMigration1.add(Value.get("Baja California"));whaleMigration1.add(Value.get("Chukchi Sea"));
ArrayList<Value> whaleMigration2 = new ArrayList<Value>();whaleMigration2.add(Value.get(1278));whaleMigration2.add(Value.get("north pacific right whale"));whaleMigration2.add(Value.get("Japan"));whaleMigration2.add(Value.get("Sea of Okhotsk"));
ArrayList<Value> whaleMigration3 = new ArrayList<Value>();whaleMigration3.add(Value.get(5100));whaleMigration3.add(Value.get("humpback whale"));whaleMigration3.add(Value.get("Columbia"));whaleMigration3.add(Value.get("Antarctic Peninsula"));
ArrayList<Value> whaleMigration4 = new ArrayList<Value>();whaleMigration4.add(Value.get(3100));whaleMigration4.add(Value.get("southern hemisphere blue whale"));whaleMigration4.add(Value.get("Corcovado Gulf"));whaleMigration4.add(Value.get("The Galapagos"));


ArrayList<Value> whaleMigration = new ArrayList<Value>();whaleMigration.add(Value.get(whaleMigration0));whaleMigration.add(Value.get(whaleMigration1));whaleMigration.add(Value.get(whaleMigration2));whaleMigration.add(Value.get(whaleMigration3));whaleMigration.add(Value.get(whaleMigration4));

// Create Map of Whale Observations
HashMap <Value, Value> mapObs = new HashMap <Value, Value>();HashMap <String, Integer> mapCoords0 = new HashMap <String, Integer>();mapCoords0.put("lat", -85);mapCoords0.put("long", -130);HashMap <String, Integer> mapCoords1 = new HashMap <String, Integer>();mapCoords1.put("lat", -25);mapCoords1.put("long", -50);HashMap <String, Integer> mapCoords2 = new HashMap <String, Integer>();mapCoords2.put("lat", 35);mapCoords2.put("long", 30);

mapObs.put(Value.get(13456), Value.get(mapCoords1));mapObs.put(Value.get(14567), Value.get(mapCoords2));mapObs.put(Value.get(12345), Value.get(mapCoords0));

// Put data in Aerospike, get the data, and print it
String nestedCDTSetName = "nestedset1";String nestedCDTNamespaceName = "test";
Integer whaleMigrationWriteFlags = ListWriteFlags.ADD_UNIQUE                                     | ListWriteFlags.NO_FAIL                                     | ListWriteFlags.PARTIAL;ListPolicy whaleMigrationPolicy = new ListPolicy(ListOrder.UNORDERED, whaleMigrationWriteFlags);MapPolicy mapObsPolicy = new MapPolicy(MapOrder.KEY_ORDERED, MapWriteFlags.DEFAULT);
Integer whaleKeyName = 2;String listWhaleBinName = "listwhalebin";String mapObsBinName = "mapobsbin";
Bin bin1 = new Bin(listWhaleBinName, whaleMigration);
Key whaleKey = new Key(nestedCDTNamespaceName, nestedCDTSetName, whaleKeyName);
Record putDataIn = client.operate(client.writePolicyDefault, whaleKey,    Operation.put(bin1),    MapOperation.putItems(mapObsPolicy, mapObsBinName, mapObs)    );
System.out.println(listWhaleBinName + ": " + whaleMigration + "\n\n" +     mapObsBinName + ": " + mapObs );

Output:

listwhalebin: [[1420, beluga whale, Beaufort Sea, Bering Sea], [13988, gray whale, Baja California, Chukchi Sea], [1278, north pacific right whale, Japan, Sea of Okhotsk], [5100, humpback whale, Columbia, Antarctic Peninsula], [3100, southern hemisphere blue whale, Corcovado Gulf, The Galapagos]]
mapobsbin: {13456={lat=-25, long=-50}, 14567={lat=35, long=30}, 12345={lat=-85, long=-130}}

Using Contexts (CTXs) to work with Nested CDTs#

What are Nested CDTs and CTXs?

What is a Nested CDT?#

The primary use case of Key-Value Stores, like Aerospike Database, is to store document-oriented data, like a JSON map. As document-oriented data grows organically, it is common for one CDT (list or map) to contain another CDT. Does the application need a list in a map in a list in a map? Aerospike fully supports nesting CDTs, so that’s no problem.

What is a Context?#

A Context (CTX) is a reference to a nested CDT, a List or Map that is stored in a List or Map somewhere in an Aerospike Bin. All List and Map Operations accept an optional CTX argument. Any CTX argument must refer to data of the type supported by the operation.

The most common ways to access a CTX are to look up a Map CTX directly by its key within the Bin and to drill down within a List or Map by index, rank or value. A CTX can also be created within a List or Map. For more details, see the CTX APIs.

Look up a Map CTX in a Bin by Mapkey#

Use the mapKey method to look up a CTX in a Map directly by mapkey. This works for a Map anywhere in a Bin.

The following is an example of finding a Map CTX in a Bin by Mapkey:

import com.aerospike.client.cdt.CTX;import com.aerospike.client.cdt.MapReturnType;
Integer lookupMapKey = 14567;String latKeyName = "lat";
Record whaleSightings = client.operate(client.writePolicyDefault, whaleKey,     MapOperation.getByKey(mapObsBinName, Value.get(latKeyName), MapReturnType.VALUE, CTX.mapKey(Value.get(lookupMapKey)))    );
System.out.println(mapObsBinName + ": " + mapObs );System.out.println("The " + latKeyName + " of sighting at timestamp " + lookupMapKey + ": " + whaleSightings.getValue(mapObsBinName));

Output:

mapobsbin: {13456={lat=-25, long=-50}, 14567={lat=35, long=30}, 12345={lat=-85, long=-130}}The lat of sighting at timestamp 14567: 35

Drill down into a List or Map#

Here are the options to drill down into a CDT.

Drilling down to a CTX in a List:

  • listIndex: Lookup list by index offset.
  • listRank: Lookup list by rank.
  • listValue: Lookup list by value.

Drilling down to a CTX in a Map:

  • mapIndex: Lookup map by index offset.
  • mapRank: Lookup map by rank.
  • mapValue: Lookup map by value.

The following is an example of drilling down within a List and Map CTX:

import com.aerospike.client.cdt.ListReturnType;
// CDT Drilldown Values
Integer drilldownIndex = 2;Integer drilldownRank = 1;Value listDrilldownValue = Value.get(whaleMigration1);Value mapDrilldownValue = Value.get(mapCoords0);
// Variables to access parts of the selected CDT.
Integer getIndex = 1;
Record theRecord = client.get(null, whaleKey);Record drilldown = client.operate(client.writePolicyDefault, whaleKey,     ListOperation.getByIndex(listWhaleBinName, getIndex, MapReturnType.VALUE, CTX.listIndex(drilldownIndex)),    ListOperation.getByIndex(listWhaleBinName, getIndex, MapReturnType.VALUE, CTX.listRank(drilldownRank)),    ListOperation.getByIndex(listWhaleBinName, getIndex, MapReturnType.VALUE, CTX.listValue(listDrilldownValue)),    MapOperation.getByIndex(mapObsBinName, getIndex, MapReturnType.VALUE, CTX.mapIndex(drilldownIndex)),    MapOperation.getByIndex(mapObsBinName, getIndex, MapReturnType.VALUE, CTX.mapRank(drilldownRank)),    MapOperation.getByIndex(mapObsBinName, getIndex, MapReturnType.VALUE, CTX.mapValue(mapDrilldownValue))    );
List<?> returnWhaleList = drilldown.getList(listWhaleBinName);List<?> returnObsList = drilldown.getList(mapObsBinName); 
System.out.println("The whale migration list is: " + theRecord.getValue(listWhaleBinName) + "\n");System.out.println("The whale name from the CTX selected by index " + drilldownIndex + ": " + returnWhaleList.get(0));System.out.println("The whale name from the CTX selected by rank " + drilldownRank + ": " + returnWhaleList.get(1));System.out.println("The whale name from the CTX selected by value " + listDrilldownValue + ": " + returnWhaleList.get(2) + "\n\n");

System.out.println("The observation map is: " + theRecord.getValue(mapObsBinName) + "\n");System.out.println("The longitude of the observation from the CTX selected by index " + drilldownIndex + ": " + returnObsList.get(0));System.out.println("The longitude of the observation from the CTX selected by rank " + drilldownRank + ": " + returnObsList.get(1));System.out.println("The longitude of the observation from the CTX selected by value " + mapDrilldownValue + ": " + returnObsList.get(2));

Output:

The whale migration list is: [[1420, beluga whale, Beaufort Sea, Bering Sea], [13988, gray whale, Baja California, Chukchi Sea], [1278, north pacific right whale, Japan, Sea of Okhotsk], [5100, humpback whale, Columbia, Antarctic Peninsula], [3100, southern hemisphere blue whale, Corcovado Gulf, The Galapagos]]
The whale name from the CTX selected by index 2: north pacific right whaleThe whale name from the CTX selected by rank 1: beluga whaleThe whale name from the CTX selected by value [13988, gray whale, Baja California, Chukchi Sea]: gray whale

The observation map is: {12345={lat=-85, long=-130}, 13456={lat=-25, long=-50}, 14567={lat=35, long=30}}
The longitude of the observation from the CTX selected by index 2: 30The longitude of the observation from the CTX selected by rank 1: -50The longitude of the observation from the CTX selected by value {lat=-85, long=-130}: -130

Create a CTX Example#

If the context for the operation does not yet exist, it can be created using the following methods.

Creating a CTX in a List or Map:

  • listIndexCreate: Create list by base list's index offset.
  • mapKeyCreate: Create map by base map's key.

The following are examples of creating a list and map CTX and then writing data to the new CTX.

ArrayList<Value> newWhaleMigration = new ArrayList<Value>();newWhaleMigration.add(Value.get(1449));newWhaleMigration.add(Value.get("sei whale"));newWhaleMigration.add(Value.get("Greenland"));newWhaleMigration.add(Value.get("Gulf of Maine"));
Integer whaleIndex = 5;
HashMap <Value, Value> mapCoords3 = new HashMap <Value, Value>();mapCoords3.put(Value.get("lat"), Value.get(95));mapCoords3.put(Value.get("long"), Value.get(110));

Integer newObsKey = 15678;

Record createCTX = client.operate(client.writePolicyDefault, whaleKey,     ListOperation.insertItems(listWhaleBinName, 0, newWhaleMigration, CTX.listIndexCreate(whaleIndex, ListOrder.UNORDERED, true)),    MapOperation.putItems(mapObsPolicy, mapObsBinName, mapCoords3, CTX.mapKeyCreate(Value.get(newObsKey), MapOrder.KEY_ORDERED))    );
Record postCreate = client.get(null, whaleKey);
System.out.println("Before, the whale migration list was: " + theRecord.getValue(listWhaleBinName) + "\n");System.out.println("After the addition, it is:" + postCreate.getValue(listWhaleBinName) + "\n\n");
System.out.println("Before, the observation map was: " + theRecord.getValue(mapObsBinName) + "\n");System.out.println("After the addition, it is: " + postCreate.getValue(mapObsBinName));

Output:

Before, the whale migration list was: [[1420, beluga whale, Beaufort Sea, Bering Sea], [13988, gray whale, Baja California, Chukchi Sea], [1278, north pacific right whale, Japan, Sea of Okhotsk], [5100, humpback whale, Columbia, Antarctic Peninsula], [3100, southern hemisphere blue whale, Corcovado Gulf, The Galapagos]]
After the addition, it is:[[1420, beluga whale, Beaufort Sea, Bering Sea], [13988, gray whale, Baja California, Chukchi Sea], [1278, north pacific right whale, Japan, Sea of Okhotsk], [5100, humpback whale, Columbia, Antarctic Peninsula], [3100, southern hemisphere blue whale, Corcovado Gulf, The Galapagos], [1449, sei whale, Greenland, Gulf of Maine]]

Before, the observation map was: {12345={lat=-85, long=-130}, 13456={lat=-25, long=-50}, 14567={lat=35, long=30}}
After the addition, it is: {12345={lat=-85, long=-130}, 13456={lat=-25, long=-50}, 14567={lat=35, long=30}, 15678={lat=95, long=110}}

Choosing the Return Type Options for CDTs#

Operations on CDTs can return different types of data, depending on the return type value specified. A return type can be combined with the INVERTED flag to return all data from the CDT that was not selected by the operation. The following are the Return Types for Lists and Maps.

Standard Return Type Options for CDTs#

Aerospike Lists and Maps both provide the following return type options.

  • COUNT: Return count of items selected.
  • INDEX: Return index offset order.
  • NONE: Do not return a result.
  • RANK: Return value order. If the list/map is not ordered, Aerospike will JIT-sort the list/map.
  • REVERSE_INDEX: Return reverse index offset order.
  • REVERSE_RANK: Return value order from a version of the list sorted from maximum to minimum value. If the list is not ordered, Aerospike will JIT-sort the list.
  • VALUE: Return value for single item read and list of values from a range read.

All indexes are 0-based, with the last element accessible by index -1.

The following is an example demonstrating each possible return type from the same operation.

ArrayList<Value> lowTuple = new ArrayList<Value>();lowTuple.add(Value.get(1400));lowTuple.add(Value.NULL);
ArrayList<Value> highTuple = new ArrayList<Value>();highTuple.add(Value.get(3500));highTuple.add(Value.NULL);
Record between1400and3500 = client.operate(client.writePolicyDefault, whaleKey,     ListOperation.getByValueRange(listWhaleBinName, Value.get(lowTuple), Value.get(highTuple),         ListReturnType.COUNT),    ListOperation.getByValueRange(listWhaleBinName, Value.get(lowTuple), Value.get(highTuple),         ListReturnType.INDEX),        ListOperation.getByValueRange(listWhaleBinName, Value.get(lowTuple), Value.get(highTuple),         ListReturnType.NONE),       ListOperation.getByValueRange(listWhaleBinName, Value.get(lowTuple), Value.get(highTuple),         ListReturnType.RANK),        ListOperation.getByValueRange(listWhaleBinName, Value.get(lowTuple), Value.get(highTuple),         ListReturnType.REVERSE_INDEX),        ListOperation.getByValueRange(listWhaleBinName, Value.get(lowTuple), Value.get(highTuple),         ListReturnType.REVERSE_RANK),        ListOperation.getByValueRange(listWhaleBinName, Value.get(lowTuple), Value.get(highTuple),         ListReturnType.VALUE)       );
List<?> returnWhaleRange = between1400and3500.getList(listWhaleBinName);
System.out.println("The current whale migration list is: " + postCreate.getValue(listWhaleBinName) + "\n");System.out.println("For the whales who migrate between 1400 and 3500 miles...");System.out.println("Return COUNT: " + returnWhaleRange.get(0));System.out.println("Return INDEX: " + returnWhaleRange.get(1));System.out.println("Return NONE: has no return value.");System.out.println("Return RANK: " + returnWhaleRange.get(2));System.out.println("Return REVERSE_INDEX: " + returnWhaleRange.get(3));System.out.println("Return REVERSE_RANK: " + returnWhaleRange.get(4));System.out.println("Return Values: " + returnWhaleRange.get(5));

Output:

The current whale migration list is: [[1420, beluga whale, Beaufort Sea, Bering Sea], [13988, gray whale, Baja California, Chukchi Sea], [1278, north pacific right whale, Japan, Sea of Okhotsk], [5100, humpback whale, Columbia, Antarctic Peninsula], [3100, southern hemisphere blue whale, Corcovado Gulf, The Galapagos], [1449, sei whale, Greenland, Gulf of Maine]]
For the whales who migrate between 1400 and 3500 miles...Return COUNT: 3Return INDEX: [0, 4, 5]Return NONE: has no return value.Return RANK: [1, 2, 3]Return REVERSE_INDEX: [5, 1, 0]Return REVERSE_RANK: [2, 3, 4]Return Values: [[1420, beluga whale, Beaufort Sea, Bering Sea], [3100, southern hemisphere blue whale, Corcovado Gulf, The Galapagos], [1449, sei whale, Greenland, Gulf of Maine]]

Additional Return Type Options for Maps#

Because Maps have a replicable key/value structure, Aerospike provides options to return mapkeys or key/value pairs, in addition to value.

  • KEY: Return key for single key read and key list for range read.
  • KEY_VALUE: Return key/value pairs for items.

The following is an example demonstrating returning a key or key/value pair.

Integer latestObsRank = -1;
Record latestWhaleObs = client.operate(client.writePolicyDefault, whaleKey,     MapOperation.getByRank(mapObsBinName, latestObsRank, MapReturnType.KEY),    MapOperation.getByRank(mapObsBinName, latestObsRank, MapReturnType.KEY_VALUE)    );
List<?> latestObs = latestWhaleObs.getList(mapObsBinName);
System.out.println("The current whale observations map is: " + postCreate.getValue(mapObsBinName) + "\n");System.out.println("For the most recent observation...");System.out.println("Return the key: " + latestObs.get(0));System.out.println("Return key/value pair: " + latestObs.get(1));

Output:

The current whale observations map is: {12345={lat=-85, long=-130}, 13456={lat=-25, long=-50}, 14567={lat=35, long=30}, 15678={lat=95, long=110}}
For the most recent observation...Return the key: 15678Return key/value pair: [15678={lat=95, long=110}]

Invert the Operation Results for CDT Operations#

Aerospike also provides the INVERTED flag for CDT operations. When INVERTED is “logical or”-ed to the return type, the flag instructs a list or map operation to return the return type data for list or Map elements that were not selected by the operation. This flag instructs an operation to act as though a logical NOT operator was applied to the entire operation.

The following is an example demonstrating inverted return values.

ArrayList<Value> lowTuple = new ArrayList<Value>();lowTuple.add(Value.get(1400));lowTuple.add(Value.NULL);
ArrayList<Value> highTuple = new ArrayList<Value>();highTuple.add(Value.get(3500));highTuple.add(Value.NULL);
Record between1400and3500 = client.operate(client.writePolicyDefault, whaleKey,     ListOperation.getByValueRange(listWhaleBinName, Value.get(lowTuple), Value.get(highTuple),         ListReturnType.COUNT | ListReturnType.INVERTED),    ListOperation.getByValueRange(listWhaleBinName, Value.get(lowTuple), Value.get(highTuple),         ListReturnType.INDEX | ListReturnType.INVERTED),        ListOperation.getByValueRange(listWhaleBinName, Value.get(lowTuple), Value.get(highTuple),         ListReturnType.NONE | ListReturnType.INVERTED),       ListOperation.getByValueRange(listWhaleBinName, Value.get(lowTuple), Value.get(highTuple),         ListReturnType.RANK | ListReturnType.INVERTED),        ListOperation.getByValueRange(listWhaleBinName, Value.get(lowTuple), Value.get(highTuple),         ListReturnType.REVERSE_INDEX | ListReturnType.INVERTED),        ListOperation.getByValueRange(listWhaleBinName, Value.get(lowTuple), Value.get(highTuple),         ListReturnType.REVERSE_RANK | ListReturnType.INVERTED),        ListOperation.getByValueRange(listWhaleBinName, Value.get(lowTuple), Value.get(highTuple),         ListReturnType.VALUE | ListReturnType.INVERTED)       );
List<?> returnWhaleRange = between1400and3500.getList(listWhaleBinName);
System.out.println("The current whale migration list is: " + postCreate.getValue(listWhaleBinName) + "\n");System.out.println("For the whales who migrate between 1400 and 3500 miles...");System.out.println("Return INVERTED COUNT: " + returnWhaleRange.get(0));System.out.println("Return INVERTED INDEX: " + returnWhaleRange.get(1));System.out.println("Return INVERTED NONE: has no return value.");System.out.println("Return INVERTED RANK: " + returnWhaleRange.get(2));System.out.println("Return INVERTED REVERSE_INDEX: " + returnWhaleRange.get(3));System.out.println("Return INVERTED REVERSE_RANK: " + returnWhaleRange.get(4));System.out.println("Return INVERTED Values: " + returnWhaleRange.get(5));

Output:

The current whale migration list is: [[1420, beluga whale, Beaufort Sea, Bering Sea], [13988, gray whale, Baja California, Chukchi Sea], [1278, north pacific right whale, Japan, Sea of Okhotsk], [5100, humpback whale, Columbia, Antarctic Peninsula], [3100, southern hemisphere blue whale, Corcovado Gulf, The Galapagos], [1449, sei whale, Greenland, Gulf of Maine]]
For the whales who migrate between 1400 and 3500 miles...Return INVERTED COUNT: 3Return INVERTED INDEX: [1, 2, 3]Return INVERTED NONE: has no return value.Return INVERTED RANK: [0, 4, 5]Return INVERTED REVERSE_INDEX: [4, 3, 2]Return INVERTED REVERSE_RANK: [5, 0, 1]Return INVERTED Values: [[13988, gray whale, Baja California, Chukchi Sea], [1278, north pacific right whale, Japan, Sea of Okhotsk], [5100, humpback whale, Columbia, Antarctic Peninsula]]

Highlighting how policies shape application transactions#

Each data type operation has a write policy which can be set per CDT write/put operation to optionally:

  • Just-in-time sort the data being operated on.
  • Apply flags that instruct Aerospike’s transaction write behavior.

Create and set a MapPolicy or ListPolicy with the proper sort and write flags to change how Aerospike processes a transaction.

MapOrder and ListOrder, Just-in-time Sorting for an Operation#

By default, Maps and Lists are stored unordered. There are explicit techniques to store a list or map in order. The Map data in this notebook is key sorted. Please refer to the code snippet creating the map data (above) for an example of this. There are examples of ordering lists in the notebook Modeling Using Lists.

Applying a MapOrder or ListOrder has performance implications on operation performance. This can be a reason to apply a MapOrder or ListOrder when working with data. To understand the relative worst-case time complexity of Aerospike operations go here for lists and here for maps.

Whether to allow duplicates in a list is a function of ListOrder.

Note: Aerospike finds that worst-case performance can be helpful in determining how to prioritize application use-cases against one another, but do not set realistic performance expectations for Aerospike Database. An example where they help is asking tough questions, like, “the worst case time complexity for operation A is X, is operation A important enough to do daily or just monthly in light of the other workloads that are more time sensitive?”

Write Flags#

The following are lists of write flags for Lists and Maps. Beneath each are example transactions.

A powerful use case for Aerospike is to group operations together into single-record atomic transactions using the Operate method. This technique is used above in this notebook. When applying transactions to data, there are common circumstances where:

  • All possible operations should be executed in a fault tolerant manner
  • Specific operation failure should cause all operations to fail

Write flags can be used in any combination, as appropriate to the application and Aerospike operation being applied.

Write Flags for all CDTs#

  • DEFAULT
    • For Lists, allow duplicate values and insertions at any index.
    • For Maps, allow map create or updates.
  • NO_FAIL: Do not raise an error if a CDT item is denied due to write flag constraints.
  • PARTIAL: Allow other valid CDT items to be committed if a CDT item is denied due to write flag constraints.

These flags provide fault tolerance to transactions. Apply some combination of the above three flags–DEFAULT, NO_FAIL, and PARTIAL–to operations by using “logical or” as demonstrated below. All other write flags set conditions for operations.

Note: Without NO_FAIL, operations that fail due to the below policies will throw either error code 24 or 26.

Default Examples#

All of the above code snippets use a Default write flag policy. These operations are unrestricted by write policies.

No Fail Examples#

All of the examples in the following sections show both an exception caused by a write flag, and then pair the demonstrated write flag with No Fail to show how the same operation can fail silently.

Partial Flag Example#

Partial is generally used only in a transaction containing operations using the No Fail write flag. Otherwise, the transaction would contain no failures to overlook. The following example are a list and map transaction combining both failing and successful map and list operations.

// create policy to apply and data to trigger operation failureInteger inBoundsIndex = 0;Integer outOfBoundsIndex = 20;
HashMap <Value, Value> mapCoords4 = new HashMap <Value, Value>();mapCoords4.put(Value.get("lat"), Value.get(0));mapCoords4.put(Value.get("long"), Value.get(0));
Integer existingObsKey = 13456;
Integer listPartialWriteFlags = ListWriteFlags.INSERT_BOUNDED                                 | ListWriteFlags.NO_FAIL                                 | ListWriteFlags.PARTIAL;ListPolicy listPartialWritePolicy = new ListPolicy(ListOrder.UNORDERED, listPartialWriteFlags);
Integer mapPartialWriteFlags = MapWriteFlags.CREATE_ONLY                                 | MapWriteFlags.NO_FAIL                                 | MapWriteFlags.PARTIAL;MapPolicy mapPartialWritePolicy = new MapPolicy(MapOrder.KEY_ORDERED, mapPartialWriteFlags);

// create fresh recordInteger partialFlagKeyName = 6;Key partialFlagKey = new Key(nestedCDTNamespaceName, nestedCDTSetName, partialFlagKeyName);
Bin bin1 = new Bin(listWhaleBinName, whaleMigration);Record putDataIn = client.operate(null, partialFlagKey,        Operation.put(bin1),        MapOperation.putItems(mapObsPolicy, mapObsBinName, mapObs)    );Record partialDataPutIn = client.get(client.writePolicyDefault, partialFlagKey);

// one failed and one successful operation for both list and mapRecord partialSuccessOp = client.operate(null, partialFlagKey,        ListOperation.insert(listPartialWritePolicy, listWhaleBinName, outOfBoundsIndex, Value.get(newWhaleMigration)),        ListOperation.set(listPartialWritePolicy, listWhaleBinName, inBoundsIndex, Value.get(newWhaleMigration)),                MapOperation.put(mapPartialWritePolicy, mapObsBinName, Value.get(existingObsKey), Value.get(mapCoords4)),        MapOperation.put(mapPartialWritePolicy, mapObsBinName, Value.get(newObsKey), Value.get(mapCoords3))    );Record partialSuccessData = client.get(client.writePolicyDefault, partialFlagKey);System.out.println ("Failed to add a 5th item.\nSucceeded at changing the first item.\n");System.out.println ("Original List: " + partialDataPutIn.getValue(listWhaleBinName) + "\n");System.out.println ("Updated List: " + partialSuccessData.getValue(listWhaleBinName) + "\n\n"); System.out.println ("Failed to modify an exiting observation.\nSucceeded at adding a new observation.\n");System.out.println ("Original Map: " + partialDataPutIn.getValue(mapObsBinName) + "\n");System.out.println ("Updated Map: " + partialSuccessData.getValue(mapObsBinName) + "\n\nFor more about the failed operations, see the examples below.");
Boolean partialExampleRecordDeleted=client.delete(null, partialFlagKey);

Output:

Failed to add a 5th item.Succeeded at changing the first item.
Original List: [[1420, beluga whale, Beaufort Sea, Bering Sea], [13988, gray whale, Baja California, Chukchi Sea], [1278, north pacific right whale, Japan, Sea of Okhotsk], [5100, humpback whale, Columbia, Antarctic Peninsula], [3100, southern hemisphere blue whale, Corcovado Gulf, The Galapagos]]
Updated List: [[1449, sei whale, Greenland, Gulf of Maine], [13988, gray whale, Baja California, Chukchi Sea], [1278, north pacific right whale, Japan, Sea of Okhotsk], [5100, humpback whale, Columbia, Antarctic Peninsula], [3100, southern hemisphere blue whale, Corcovado Gulf, The Galapagos]]

Failed to modify an exiting observation.Succeeded at adding a new observation.
Original Map: {12345={lat=-85, long=-130}, 13456={lat=-25, long=-50}, 14567={lat=35, long=30}}
Updated Map: {12345={lat=-85, long=-130}, 13456={lat=-25, long=-50}, 14567={lat=35, long=30}, 15678={lat=95, long=110}}
For more about the failed operations, see the examples below.

Write Flags for Lists Only:#

  • INSERT_BOUNDED: Enforce list boundaries when inserting. Do not allow values to be inserted at index outside current list boundaries.
  • ADD_UNIQUE: Only add unique values.

Insert Bounded Example#

// create policy to apply and data to break policyInteger outOfBoundsIndex = 20;
ListPolicy listInsertBoundedPolicy = new ListPolicy(ListOrder.UNORDERED, ListWriteFlags.INSERT_BOUNDED);ListPolicy listBoundedNoFailPolicy = new ListPolicy(ListOrder.UNORDERED, ListWriteFlags.INSERT_BOUNDED                                                                             | ListWriteFlags.NO_FAIL);
// create fresh recordInteger whaleBoundedKeyName = 7;
Bin bin1 = new Bin(listWhaleBinName, whaleMigration);
Key whaleBoundedKey = new Key(nestedCDTNamespaceName, nestedCDTSetName, whaleBoundedKeyName);
client.put(client.writePolicyDefault, whaleBoundedKey, bin1);Record ibDataPutIn = client.get(null, whaleBoundedKey);System.out.println("Data in the record: " + ibDataPutIn.getValue(listWhaleBinName) + "\n");

// fail for INSERT_BOUNDEDtry {    Record ibFail = client.operate(client.writePolicyDefault, whaleBoundedKey,        ListOperation.insert(listInsertBoundedPolicy, listWhaleBinName, outOfBoundsIndex, Value.get(newWhaleMigration))    );    System.out.println("The code does not get here.");} catch(Exception e) {    System.out.println("Out of Bounds Attempt 1: Exception caught.");    Record ibNoFail = client.operate(client.writePolicyDefault, whaleBoundedKey,        ListOperation.insert(listBoundedNoFailPolicy, listWhaleBinName, outOfBoundsIndex, Value.get(newWhaleMigration))    );    Record ibNoFailData = client.get(client.writePolicyDefault, whaleBoundedKey);    if(ibNoFailData.getValue(listWhaleBinName).equals(ibDataPutIn.getValue(listWhaleBinName))) {        System.out.println("Out of Bounds Attempt 2: No operation was executed. Error was suppressed by NO_FAIL.\n");    }}
Record noIB = client.operate(client.writePolicyDefault, whaleBoundedKey,        ListOperation.insert(listWhaleBinName, outOfBoundsIndex, Value.get(newWhaleMigration)));Record noIBData = client.get(null, whaleBoundedKey);System.out.println("Without Insert Bounded, a series of nulls is insein the Bin: " + noIBData.getValue(listWhaleBinName));

Output:

Data in the record: [[1420, beluga whale, Beaufort Sea, Bering Sea], [13988, gray whale, Baja California, Chukchi Sea], [1278, north pacific right whale, Japan, Sea of Okhotsk], [5100, humpback whale, Columbia, Antarctic Peninsula], [3100, southern hemisphere blue whale, Corcovado Gulf, The Galapagos]]
Out of Bounds Attempt 1: Exception caught.Out of Bounds Attempt 2: No operation was executed. Error was suppressed by NO_FAIL.
Without Insert Bounded, a series of nulls is insein the Bin: [[1420, beluga whale, Beaufort Sea, Bering Sea], [13988, gray whale, Baja California, Chukchi Sea], [1278, north pacific right whale, Japan, Sea of Okhotsk], [5100, humpback whale, Columbia, Antarctic Peninsula], [3100, southern hemisphere blue whale, Corcovado Gulf, The Galapagos], null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, [1449, sei whale, Greenland, Gulf of Maine]]

Add Unique Example#

// create policy to applyListPolicy listAddUniquePolicy = new ListPolicy(ListOrder.UNORDERED, ListWriteFlags.ADD_UNIQUE);ListPolicy listAddUniqueNoFailPolicy = new ListPolicy(ListOrder.UNORDERED, ListWriteFlags.ADD_UNIQUE                                                                             | ListWriteFlags.NO_FAIL);
// create fresh recordInteger whaleAddUniqueKeyName = 8;Bin bin1 = new Bin(listWhaleBinName, whaleMigration);Key whaleAddUniqueKey = new Key(nestedCDTNamespaceName, nestedCDTSetName, whaleAddUniqueKeyName);client.put(client.writePolicyDefault, whaleAddUniqueKey, bin1);Record auDataPutIn = client.get(null, whaleAddUniqueKey);

// successful ADD_UNIQUE operationRecord auSuccess = client.operate(client.writePolicyDefault, whaleAddUniqueKey,    ListOperation.append(listAddUniquePolicy, listWhaleBinName, Value.get(newWhaleMigration)));Record auSuccessData = client.get(null, whaleAddUniqueKey);
System.out.println("Data after the unique add of " + newWhaleMigration + ": " + auSuccessData.getValue(listWhaleBinName) + "\n");

// fail for 2nd ADD_UNIQUEtry {    Record auFail = client.operate(client.writePolicyDefault, whaleAddUniqueKey,        ListOperation.append(listAddUniquePolicy, listWhaleBinName, Value.get(newWhaleMigration))    );    System.out.println("The code does not get here.");}catch(Exception e) {    System.out.println("Non-Unique Add 1: Exception caught.");    Record auNoFail = client.operate(client.writePolicyDefault, whaleAddUniqueKey,        ListOperation.append(listAddUniqueNoFailPolicy, listWhaleBinName, Value.get(newWhaleMigration))    );    Record auNoFailData = client.get(null, whaleAddUniqueKey);    if(auNoFailData.getValue(listWhaleBinName).equals(auSuccessData.getValue(listWhaleBinName))) {        System.out.println("Non-Unique Add 2: No operation was executed. Error was suppressed by NO_FAIL.\n");    }}
Record noAU = client.operate(client.writePolicyDefault, whaleAddUniqueKey,        ListOperation.append(listWhaleBinName, Value.get(newWhaleMigration)));Record noAUData = client.get(null, whaleAddUniqueKey);System.out.println("Without Add Unique here, the tuple for a sei whale is there 2x: " + noAUData.getValue(listWhaleBinName));

Output:

Data after the unique add of [1449, sei whale, Greenland, Gulf of Maine]: [[1420, beluga whale, Beaufort Sea, Bering Sea], [13988, gray whale, Baja California, Chukchi Sea], [1278, north pacific right whale, Japan, Sea of Okhotsk], [5100, humpback whale, Columbia, Antarctic Peninsula], [3100, southern hemisphere blue whale, Corcovado Gulf, The Galapagos], [1449, sei whale, Greenland, Gulf of Maine]]
Non-Unique Add 1: Exception caught.Non-Unique Add 2: No operation was executed. Error was suppressed by NO_FAIL.
Without Add Unique here, the tuple for a sei whale is there 2x: [[1420, beluga whale, Beaufort Sea, Bering Sea], [13988, gray whale, Baja California, Chukchi Sea], [1278, north pacific right whale, Japan, Sea of Okhotsk], [5100, humpback whale, Columbia, Antarctic Peninsula], [3100, southern hemisphere blue whale, Corcovado Gulf, The Galapagos], [1449, sei whale, Greenland, Gulf of Maine], [1449, sei whale, Greenland, Gulf of Maine]]

Write Flags for Maps Only:#

  • CREATE_ONLY: If the key already exists, the item will be denied.
  • UPDATE_ONLY: If the key already exists, the item will be overwritten. If the key does not exist, the item will be denied.

Create Only Example#

// create modify data and policy to applyHashMap <Value, Value> mapCoords4 = new HashMap <Value, Value>();mapCoords4.put(Value.get("lat"), Value.get(0));mapCoords4.put(Value.get("long"), Value.get(0));
MapPolicy mapCreateOnlyPolicy = new MapPolicy(MapOrder.KEY_ORDERED, MapWriteFlags.CREATE_ONLY);MapPolicy mapCreateOnlyNoFailPolicy = new MapPolicy(MapOrder.KEY_ORDERED, MapWriteFlags.CREATE_ONLY                                                                             | MapWriteFlags.NO_FAIL);
// create fresh recordInteger obsCreateOnlyKeyName = 9;Key obsCreateOnlyKey = new Key(nestedCDTNamespaceName, nestedCDTSetName, obsCreateOnlyKeyName);Record putDataIn = client.operate(client.writePolicyDefault, obsCreateOnlyKey,    MapOperation.putItems(mapObsPolicy, mapObsBinName, mapObs)    );Record coDataPutIn = client.get(null, obsCreateOnlyKey);

// success for CREATE_ONLYRecord coSuccess = client.operate(client.writePolicyDefault, obsCreateOnlyKey,    MapOperation.put(mapCreateOnlyPolicy, mapObsBinName, Value.get(newObsKey), Value.get(mapCoords3)));Record coSuccessData = client.get(null, obsCreateOnlyKey);System.out.println("Created record and new key " + newObsKey + ". The data is now: " + coSuccessData.getValue(mapObsBinName) + "\n");

// fail for CREATE_ONLYtry {    Record coFail = client.operate(client.writePolicyDefault, obsCreateOnlyKey,        MapOperation.put(mapCreateOnlyPolicy, mapObsBinName, Value.get(newObsKey), Value.get(mapCoords4))    );    System.out.println("The code does not get here.");}catch(Exception e) {    System.out.println("Update attempt 1: Exception caught.");    Record coNoFail = client.operate(client.writePolicyDefault, obsCreateOnlyKey,        MapOperation.put(mapCreateOnlyNoFailPolicy, mapObsBinName, Value.get(newObsKey), Value.get(mapCoords4))    );    Record coNoFailData = client.get(null, obsCreateOnlyKey);    if(coNoFailData.getValue(mapObsBinName).equals(coSuccessData.getValue(mapObsBinName))) {        System.out.println("Update attempt 2: No operation was executed. Error was suppressed by NO_FAIL.\n");    }}
Record noCO = client.operate(client.writePolicyDefault, obsCreateOnlyKey,     MapOperation.put(mapObsPolicy, mapObsBinName, Value.get(newObsKey), Value.get(mapCoords4)));Record noCOData = client.get(null, obsCreateOnlyKey);System.out.println("Without Create Only, the observation at 15678 is overwritten: " + noCOData.getValue(mapObsBinName));
Boolean createOnlyExampleRecordDeleted=client.delete(null, obsCreateOnlyKey);

Output:

Created record and new key 15678. The data is now: {12345={lat=-85, long=-130}, 13456={lat=-25, long=-50}, 14567={lat=35, long=30}, 15678={lat=95, long=110}}
Update attempt 1: Exception caught.Update attempt 2: No operation was executed. Error was suppressed by NO_FAIL.
Without Create Only, the observation at 15678 is overwritten: {12345={lat=-85, long=-130}, 13456={lat=-25, long=-50}, 14567={lat=35, long=30}, 15678={lat=0, long=0}}

Update Only Example#

// create policy to applyMapPolicy mapUpdateOnlyPolicy = new MapPolicy(MapOrder.KEY_ORDERED, MapWriteFlags.UPDATE_ONLY);MapPolicy mapUpdateOnlyNoFailPolicy = new MapPolicy(MapOrder.KEY_ORDERED, MapWriteFlags.UPDATE_ONLY                                                                             | MapWriteFlags.NO_FAIL);
// create Aerospike data elements for a fresh recordInteger obsUpdateOnlyKeyName = 10;Key obsUpdateOnlyKey = new Key(nestedCDTNamespaceName, nestedCDTSetName, obsUpdateOnlyKeyName);
Record uoPutDataIn = client.operate(client.writePolicyDefault, obsUpdateOnlyKey,    MapOperation.putItems(mapObsPolicy, mapObsBinName, mapObs)    );Record uoDataPutIn = client.get(null, obsUpdateOnlyKey);System.out.println("Created record: " + uoDataPutIn.getValue(mapObsBinName) + "\n");

// fail for UPDATE_ONLYtry {    Record uoFail = client.operate(client.writePolicyDefault, obsUpdateOnlyKey,        MapOperation.put(mapUpdateOnlyPolicy, mapObsBinName, Value.get(newObsKey), Value.get(mapCoords3))    );    System.out.println("The code does not get here.");}catch(Exception e) {    System.out.println("Create Attempt 1: Exception caught.");    Record uoNoFail = client.operate(client.writePolicyDefault, obsUpdateOnlyKey,        MapOperation.put(mapUpdateOnlyNoFailPolicy, mapObsBinName, Value.get(newObsKey), Value.get(mapCoords3))    );    Record uoNoFailData = client.get(null, obsUpdateOnlyKey);    if(uoNoFailData.getValue(mapObsBinName).equals(uoDataPutIn.getValue(mapObsBinName))){        System.out.println("Create Attempt 2: No operation was executed. Error was suppressed by NO_FAIL.\n");    }}
Record noUO = client.operate(client.writePolicyDefault, obsUpdateOnlyKey,         MapOperation.put(mapObsPolicy, mapObsBinName, Value.get(newObsKey), Value.get(mapCoords3)));Record noUOData = client.get(null, obsUpdateOnlyKey);
// success for UPDATE_ONLYRecord uoSuccess = client.operate(client.writePolicyDefault, obsUpdateOnlyKey,    MapOperation.put(mapUpdateOnlyPolicy, mapObsBinName, Value.get(existingObsKey), Value.get(mapCoords4)));Record uoSuccessData = client.get(null, obsUpdateOnlyKey);System.out.println("Using update only, the value of an existing key " + existingObsKey + " can be updated: " + uoSuccessData.getValue(mapObsBinName) + "\n");
Boolean uoExampleRecordDeleted=client.delete(null, obsUpdateOnlyKey);

Output:

Created record: {12345={lat=-85, long=-130}, 13456={lat=-25, long=-50}, 14567={lat=35, long=30}}
Create Attempt 1: Exception caught.Create Attempt 2: No operation was executed. Error was suppressed by NO_FAIL.
Using update only, the value of an existing key 13456 can be updated: {12345={lat=-85, long=-130}, 13456={lat=0, long=0}, 14567={lat=35, long=30}, 15678={lat=95, long=110}}

Notebook Cleanup#

Truncate the Set#

Truncate the set from the Aerospike Database.

import com.aerospike.client.policy.InfoPolicy;InfoPolicy infoPolicy = new InfoPolicy();
client.truncate(infoPolicy, nestedCDTNamespaceName, nestedCDTSetName, null);System.out.println("Set Truncated.");

Output:

Set Truncated.

Close the Client connections to Aerospike#

client.close();System.out.println("Server connection(s) closed.");

Output:

Server connection(s) closed.

Takeaways – CDTs Provide Flexible Document-Oriented Data Power#

Aerospike Collection Data Types...

  1. facilitate complex data structures by supporting nesting through the use of contexts (CTXs)
  2. provide intuitive and flexible return types options from operations
  3. support policies that empower efficient and flexible transaction processing

What's Next?#

Next Steps#

Have questions? Don't hesitate to reach out if you have additional questions about data modeling at https://discuss.aerospike.com/c/how-developers-are-using-aerospike/data-modeling/143.

Want to check out other Java notebooks?

  1. Intro to Transactions
  2. Modeling Using Lists
  3. Working with Maps
  4. Aerospike Query and UDF

Are you running this from Binder? Download the Aerospike Notebook Repo and work with Aerospike Database and Jupyter locally using a Docker container.

Additional Resources#