Advanced Collection Data Types

For an interactive Jupyter notebook experience:

Last updated: June 22, 2021

The goal of this tutorial is to highlight the power of working with collection data types (CDTs) in Aerospike. It covers the following topics:

Setting contexts (CTXs) to apply operations to nested Maps and Lists.
Showing the return type options provided by CDT get/read operations.
Highlighting how policies shape application transactions.

This Jupyter Notebook requires the Aerospike Database running locally with Java kernel and Aerospike Java Client. To create a Docker container that satisfies the requirements and holds a copy of these notebooks, visit the Aerospike Notebooks Repo.

Prerequisites

This Notebook builds on the material in the following notebooks:

It uses examples based on those from Modeling Using Lists and Working with Maps. If any of the following is confusing, please refer to a relevant notebook.

Notebook Setup

Import Jupyter Java Integration

Make it easier to work with Java in Jupyter.

import io.github.spencerpark.ijava.IJava;
import io.github.spencerpark.jupyter.kernel.magic.common.Shell;

IJava.getKernelInstance().getMagics().registerMagics(Shell.class);

Start Aerospike

Ensure Aerospike Database is running locally.

%sh asd

Download the Aerospike Java Client

Ask Maven to download and install the project object model (POM) of the Aerospike Java Client.

%%loadFromPOM
<dependencies>
  <dependency>
    <groupId>com.aerospike</groupId>
    <artifactId>aerospike-client</artifactId>
    <version>5.0.0</version>
  </dependency>
</dependencies>

Start the Aerospike Java Client and Connect

Create an instance of the Aerospike Java Client, and connect to the demo cluster.

The default cluster location for the Docker container is localhost port 3000. If your cluster is not running on your local machine, modify localhost and 3000 to the values for your Aerospike cluster.

import com.aerospike.client.AerospikeClient;

AerospikeClient client = new AerospikeClient("localhost", 3000);
System.out.println("Initialized the client and connected to the cluster.");

Output:

Initialized the client and connected to the cluster.

Create CDT Data, Put into Aerospike, and Print It

import com.aerospike.client.Key;
import com.aerospike.client.Bin;
import com.aerospike.client.policy.ClientPolicy;
import com.aerospike.client.Record;
import com.aerospike.client.Operation;
import com.aerospike.client.Value;
import com.aerospike.client.cdt.ListOperation;
import com.aerospike.client.cdt.ListPolicy;
import com.aerospike.client.cdt.ListOrder;
import com.aerospike.client.cdt.ListWriteFlags;
import com.aerospike.client.cdt.MapOperation;
import com.aerospike.client.cdt.MapPolicy;
import com.aerospike.client.cdt.MapOrder;
import com.aerospike.client.cdt.MapWriteFlags;


import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;


// Create whale migration list of tuples. 

ArrayList<Value> whaleMigration0 = new ArrayList<Value>();
whaleMigration0.add(Value.get(1420));
whaleMigration0.add(Value.get("beluga whale"));
whaleMigration0.add(Value.get("Beaufort Sea"));
whaleMigration0.add(Value.get("Bering Sea"));

ArrayList<Value> whaleMigration1 = new ArrayList<Value>();
whaleMigration1.add(Value.get(13988));
whaleMigration1.add(Value.get("gray whale"));
whaleMigration1.add(Value.get("Baja California"));
whaleMigration1.add(Value.get("Chukchi Sea"));

ArrayList<Value> whaleMigration2 = new ArrayList<Value>();
whaleMigration2.add(Value.get(1278));
whaleMigration2.add(Value.get("north pacific right whale"));
whaleMigration2.add(Value.get("Japan"));
whaleMigration2.add(Value.get("Sea of Okhotsk"));

ArrayList<Value> whaleMigration3 = new ArrayList<Value>();
whaleMigration3.add(Value.get(5100));
whaleMigration3.add(Value.get("humpback whale"));
whaleMigration3.add(Value.get("Columbia"));
whaleMigration3.add(Value.get("Antarctic Peninsula"));

ArrayList<Value> whaleMigration4 = new ArrayList<Value>();
whaleMigration4.add(Value.get(3100));
whaleMigration4.add(Value.get("southern hemisphere blue whale"));
whaleMigration4.add(Value.get("Corcovado Gulf"));
whaleMigration4.add(Value.get("The Galapagos"));



ArrayList<Value> whaleMigration = new ArrayList<Value>();
whaleMigration.add(Value.get(whaleMigration0));
whaleMigration.add(Value.get(whaleMigration1));
whaleMigration.add(Value.get(whaleMigration2));
whaleMigration.add(Value.get(whaleMigration3));
whaleMigration.add(Value.get(whaleMigration4));


// Create Map of Whale Observations

HashMap <Value, Value> mapObs = new HashMap <Value, Value>();
HashMap <String, Integer> mapCoords0 = new HashMap <String, Integer>();
mapCoords0.put("lat", -85);
mapCoords0.put("long", -130);
HashMap <String, Integer> mapCoords1 = new HashMap <String, Integer>();
mapCoords1.put("lat", -25);
mapCoords1.put("long", -50);
HashMap <String, Integer> mapCoords2 = new HashMap <String, Integer>();
mapCoords2.put("lat", 35);
mapCoords2.put("long", 30);


mapObs.put(Value.get(13456), Value.get(mapCoords1));
mapObs.put(Value.get(14567), Value.get(mapCoords2));
mapObs.put(Value.get(12345), Value.get(mapCoords0));


// Put data in Aerospike, get the data, and print it

String nestedCDTSetName = "nestedset1";
String nestedCDTNamespaceName = "test";

Integer whaleMigrationWriteFlags = ListWriteFlags.ADD_UNIQUE 
                                    | ListWriteFlags.NO_FAIL 
                                    | ListWriteFlags.PARTIAL;
ListPolicy whaleMigrationPolicy = new ListPolicy(ListOrder.UNORDERED, whaleMigrationWriteFlags);
MapPolicy mapObsPolicy = new MapPolicy(MapOrder.KEY_ORDERED, MapWriteFlags.DEFAULT);

Integer whaleKeyName = 2;
String listWhaleBinName = "listwhalebin";
String mapObsBinName = "mapobsbin";

Bin bin1 = new Bin(listWhaleBinName, whaleMigration);

Key whaleKey = new Key(nestedCDTNamespaceName, nestedCDTSetName, whaleKeyName);

Record putDataIn = client.operate(client.writePolicyDefault, whaleKey,
    Operation.put(bin1),
    MapOperation.putItems(mapObsPolicy, mapObsBinName, mapObs)
    );

System.out.println(listWhaleBinName + ": " + whaleMigration + "\n\n" + 
    mapObsBinName + ": " + mapObs );

Output:

listwhalebin: [[1420, beluga whale, Beaufort Sea, Bering Sea], [13988, gray whale, Baja California, Chukchi Sea], [1278, north pacific right whale, Japan, Sea of Okhotsk], [5100, humpback whale, Columbia, Antarctic Peninsula], [3100, southern hemisphere blue whale, Corcovado Gulf, The Galapagos]]

mapobsbin: {13456={lat=-25, long=-50}, 14567={lat=35, long=30}, 12345={lat=-85, long=-130}}

Using Contexts (CTXs) to work with Nested CDTs

What are Nested CDTs and CTXs?

What is a Nested CDT?

The primary use case of Key-Value Stores, like Aerospike Database, is to store document-oriented data, like a JSON map. As document-oriented data grows organically, it is common for one CDT (list or map) to contain another CDT. Does the application need a list in a map in a list in a map? Aerospike fully supports nesting CDTs, so that’s no problem.

What is a Context?

A Context (CTX) is a reference to a nested CDT, a List or Map that is stored in a List or Map somewhere in an Aerospike Bin. All List and Map Operations accept an optional CTX argument. Any CTX argument must refer to data of the type supported by the operation.

The most common ways to access a CTX are to look up a Map CTX directly by its key within the Bin and to drill down within a List or Map by index, rank or value. A CTX can also be created within a List or Map. For more details, see the CTX APIs.

Look up a Map CTX in a Bin by Mapkey

Use the mapKey method to look up a CTX in a Map directly by mapkey. This works for a Map anywhere in a Bin.

The following is an example of finding a Map CTX in a Bin by Mapkey:

import com.aerospike.client.cdt.CTX;
import com.aerospike.client.cdt.MapReturnType;

Integer lookupMapKey = 14567;
String latKeyName = "lat";

Record whaleSightings = client.operate(client.writePolicyDefault, whaleKey, 
    MapOperation.getByKey(mapObsBinName, Value.get(latKeyName), MapReturnType.VALUE, CTX.mapKey(Value.get(lookupMapKey)))
    );

System.out.println(mapObsBinName + ": " + mapObs );
System.out.println("The " + latKeyName + " of sighting at timestamp " + lookupMapKey + ": " + whaleSightings.getValue(mapObsBinName));

Output:

mapobsbin: {13456={lat=-25, long=-50}, 14567={lat=35, long=30}, 12345={lat=-85, long=-130}}
The lat of sighting at timestamp 14567: 35

Drill down into a List or Map

Here are the options to drill down into a CDT.

Drilling down to a CTX in a List:

listIndex: Lookup list by index offset.
listRank: Lookup list by rank.
listValue: Lookup list by value.

Drilling down to a CTX in a Map:

mapIndex: Lookup map by index offset.
mapRank: Lookup map by rank.
mapValue: Lookup map by value.

The following is an example of drilling down within a List and Map CTX:

import com.aerospike.client.cdt.ListReturnType;

// CDT Drilldown Values

Integer drilldownIndex = 2;
Integer drilldownRank = 1;
Value listDrilldownValue = Value.get(whaleMigration1);
Value mapDrilldownValue = Value.get(mapCoords0);

// Variables to access parts of the selected CDT.

Integer getIndex = 1;

Record theRecord = client.get(null, whaleKey);
Record drilldown = client.operate(client.writePolicyDefault, whaleKey, 
    ListOperation.getByIndex(listWhaleBinName, getIndex, MapReturnType.VALUE, CTX.listIndex(drilldownIndex)),
    ListOperation.getByIndex(listWhaleBinName, getIndex, MapReturnType.VALUE, CTX.listRank(drilldownRank)),
    ListOperation.getByIndex(listWhaleBinName, getIndex, MapReturnType.VALUE, CTX.listValue(listDrilldownValue)),
    MapOperation.getByIndex(mapObsBinName, getIndex, MapReturnType.VALUE, CTX.mapIndex(drilldownIndex)),
    MapOperation.getByIndex(mapObsBinName, getIndex, MapReturnType.VALUE, CTX.mapRank(drilldownRank)),
    MapOperation.getByIndex(mapObsBinName, getIndex, MapReturnType.VALUE, CTX.mapValue(mapDrilldownValue))
    );

List<?> returnWhaleList = drilldown.getList(listWhaleBinName);
List<?> returnObsList = drilldown.getList(mapObsBinName); 

System.out.println("The whale migration list is: " + theRecord.getValue(listWhaleBinName) + "\n");
System.out.println("The whale name from the CTX selected by index " + drilldownIndex + ": " + returnWhaleList.get(0));
System.out.println("The whale name from the CTX selected by rank " + drilldownRank + ": " + returnWhaleList.get(1));
System.out.println("The whale name from the CTX selected by value " + listDrilldownValue + ": " + returnWhaleList.get(2) + "\n\n");


System.out.println("The observation map is: " + theRecord.getValue(mapObsBinName) + "\n");
System.out.println("The longitude of the observation from the CTX selected by index " + drilldownIndex + ": " + returnObsList.get(0));
System.out.println("The longitude of the observation from the CTX selected by rank " + drilldownRank + ": " + returnObsList.get(1));
System.out.println("The longitude of the observation from the CTX selected by value " + mapDrilldownValue + ": " + returnObsList.get(2));

Output:

The whale migration list is: [[1420, beluga whale, Beaufort Sea, Bering Sea], [13988, gray whale, Baja California, Chukchi Sea], [1278, north pacific right whale, Japan, Sea of Okhotsk], [5100, humpback whale, Columbia, Antarctic Peninsula], [3100, southern hemisphere blue whale, Corcovado Gulf, The Galapagos]]

The whale name from the CTX selected by index 2: north pacific right whale
The whale name from the CTX selected by rank 1: beluga whale
The whale name from the CTX selected by value [13988, gray whale, Baja California, Chukchi Sea]: gray whale


The observation map is: {12345={lat=-85, long=-130}, 13456={lat=-25, long=-50}, 14567={lat=35, long=30}}

The longitude of the observation from the CTX selected by index 2: 30
The longitude of the observation from the CTX selected by rank 1: -50
The longitude of the observation from the CTX selected by value {lat=-85, long=-130}: -130

Create a CTX Example

If the context for the operation does not yet exist, it can be created using the following methods.

Creating a CTX in a List or Map:

listIndexCreate: Create list by base list's index offset.
mapKeyCreate: Create map by base map's key.

The following are examples of creating a list and map CTX and then writing data to the new CTX.

ArrayList<Value> newWhaleMigration = new ArrayList<Value>();
newWhaleMigration.add(Value.get(1449));
newWhaleMigration.add(Value.get("sei whale"));
newWhaleMigration.add(Value.get("Greenland"));
newWhaleMigration.add(Value.get("Gulf of Maine"));

Integer whaleIndex = 5;

HashMap <Value, Value> mapCoords3 = new HashMap <Value, Value>();
mapCoords3.put(Value.get("lat"), Value.get(95));
mapCoords3.put(Value.get("long"), Value.get(110));


Integer newObsKey = 15678;


Record createCTX = client.operate(client.writePolicyDefault, whaleKey, 
    ListOperation.insertItems(listWhaleBinName, 0, newWhaleMigration, CTX.listIndexCreate(whaleIndex, ListOrder.UNORDERED, true)),
    MapOperation.putItems(mapObsPolicy, mapObsBinName, mapCoords3, CTX.mapKeyCreate(Value.get(newObsKey), MapOrder.KEY_ORDERED))
    );

Record postCreate = client.get(null, whaleKey);

System.out.println("Before, the whale migration list was: " + theRecord.getValue(listWhaleBinName) + "\n");
System.out.println("After the addition, it is:" + postCreate.getValue(listWhaleBinName) + "\n\n");

System.out.println("Before, the observation map was: " + theRecord.getValue(mapObsBinName) + "\n");
System.out.println("After the addition, it is: " + postCreate.getValue(mapObsBinName));

Output:

Before, the whale migration list was: [[1420, beluga whale, Beaufort Sea, Bering Sea], [13988, gray whale, Baja California, Chukchi Sea], [1278, north pacific right whale, Japan, Sea of Okhotsk], [5100, humpback whale, Columbia, Antarctic Peninsula], [3100, southern hemisphere blue whale, Corcovado Gulf, The Galapagos]]

After the addition, it is:[[1420, beluga whale, Beaufort Sea, Bering Sea], [13988, gray whale, Baja California, Chukchi Sea], [1278, north pacific right whale, Japan, Sea of Okhotsk], [5100, humpback whale, Columbia, Antarctic Peninsula], [3100, southern hemisphere blue whale, Corcovado Gulf, The Galapagos], [1449, sei whale, Greenland, Gulf of Maine]]

Before, the observation map was: {12345={lat=-85, long=-130}, 13456={lat=-25, long=-50}, 14567={lat=35, long=30}}

After the addition, it is: {12345={lat=-85, long=-130}, 13456={lat=-25, long=-50}, 14567={lat=35, long=30}, 15678={lat=95, long=110}}

Choosing the Return Type Options for CDTs

Operations on CDTs can return different types of data, depending on the return type value specified. A return type can be combined with the INVERTED flag to return all data from the CDT that was not selected by the operation. The following are the Return Types for Lists and Maps.

Standard Return Type Options for CDTs

Aerospike Lists and Maps both provide the following return type options.

COUNT: Return count of items selected.
INDEX: Return index offset order.
NONE: Do not return a result.
RANK: Return value order. If the list/map is not ordered, Aerospike will JIT-sort the list/map.
REVERSE_INDEX: Return reverse index offset order.
REVERSE_RANK: Return value order from a version of the list sorted from maximum to minimum value. If the list is not ordered, Aerospike will JIT-sort the list.
VALUE: Return value for single item read and list of values from a range read.

All indexes are 0-based, with the last element accessible by index -1.

The following is an example demonstrating each possible return type from the same operation.

ArrayList<Value> lowTuple = new ArrayList<Value>();
lowTuple.add(Value.get(1400));
lowTuple.add(Value.NULL);

ArrayList<Value> highTuple = new ArrayList<Value>();
highTuple.add(Value.get(3500));
highTuple.add(Value.NULL);

Record between1400and3500 = client.operate(client.writePolicyDefault, whaleKey, 
    ListOperation.getByValueRange(listWhaleBinName, Value.get(lowTuple), Value.get(highTuple), 
        ListReturnType.COUNT),
    ListOperation.getByValueRange(listWhaleBinName, Value.get(lowTuple), Value.get(highTuple), 
        ListReturnType.INDEX),    
    ListOperation.getByValueRange(listWhaleBinName, Value.get(lowTuple), Value.get(highTuple), 
        ListReturnType.NONE),   
    ListOperation.getByValueRange(listWhaleBinName, Value.get(lowTuple), Value.get(highTuple), 
        ListReturnType.RANK),    
    ListOperation.getByValueRange(listWhaleBinName, Value.get(lowTuple), Value.get(highTuple), 
        ListReturnType.REVERSE_INDEX),    
    ListOperation.getByValueRange(listWhaleBinName, Value.get(lowTuple), Value.get(highTuple), 
        ListReturnType.REVERSE_RANK),    
    ListOperation.getByValueRange(listWhaleBinName, Value.get(lowTuple), Value.get(highTuple), 
        ListReturnType.VALUE)   
    );

List<?> returnWhaleRange = between1400and3500.getList(listWhaleBinName);

System.out.println("The current whale migration list is: " + postCreate.getValue(listWhaleBinName) + "\n");
System.out.println("For the whales who migrate between 1400 and 3500 miles...");
System.out.println("Return COUNT: " + returnWhaleRange.get(0));
System.out.println("Return INDEX: " + returnWhaleRange.get(1));
System.out.println("Return NONE: has no return value.");
System.out.println("Return RANK: " + returnWhaleRange.get(2));
System.out.println("Return REVERSE_INDEX: " + returnWhaleRange.get(3));
System.out.println("Return REVERSE_RANK: " + returnWhaleRange.get(4));
System.out.println("Return Values: " + returnWhaleRange.get(5));

Output:

The current whale migration list is: [[1420, beluga whale, Beaufort Sea, Bering Sea], [13988, gray whale, Baja California, Chukchi Sea], [1278, north pacific right whale, Japan, Sea of Okhotsk], [5100, humpback whale, Columbia, Antarctic Peninsula], [3100, southern hemisphere blue whale, Corcovado Gulf, The Galapagos], [1449, sei whale, Greenland, Gulf of Maine]]

For the whales who migrate between 1400 and 3500 miles...
Return COUNT: 3
Return INDEX: [0, 4, 5]
Return NONE: has no return value.
Return RANK: [1, 2, 3]
Return REVERSE_INDEX: [5, 1, 0]
Return REVERSE_RANK: [2, 3, 4]
Return Values: [[1420, beluga whale, Beaufort Sea, Bering Sea], [3100, southern hemisphere blue whale, Corcovado Gulf, The Galapagos], [1449, sei whale, Greenland, Gulf of Maine]]

Additional Return Type Options for Maps

Because Maps have a replicable key/value structure, Aerospike provides options to return mapkeys or key/value pairs, in addition to value.

KEY: Return key for single key read and key list for range read.
KEY_VALUE: Return key/value pairs for items.

The following is an example demonstrating returning a key or key/value pair.

Integer latestObsRank = -1;

Record latestWhaleObs = client.operate(client.writePolicyDefault, whaleKey, 
    MapOperation.getByRank(mapObsBinName, latestObsRank, MapReturnType.KEY),
    MapOperation.getByRank(mapObsBinName, latestObsRank, MapReturnType.KEY_VALUE)
    );

List<?> latestObs = latestWhaleObs.getList(mapObsBinName);

System.out.println("The current whale observations map is: " + postCreate.getValue(mapObsBinName) + "\n");
System.out.println("For the most recent observation...");
System.out.println("Return the key: " + latestObs.get(0));
System.out.println("Return key/value pair: " + latestObs.get(1));

Output:

The current whale observations map is: {12345={lat=-85, long=-130}, 13456={lat=-25, long=-50}, 14567={lat=35, long=30}, 15678={lat=95, long=110}}

For the most recent observation...
Return the key: 15678
Return key/value pair: [15678={lat=95, long=110}]

Invert the Operation Results for CDT Operations

Aerospike also provides the INVERTED flag for CDT operations. When INVERTED is “logical or”-ed to the return type, the flag instructs a list or map operation to return the return type data for list or Map elements that were not selected by the operation. This flag instructs an operation to act as though a logical NOT operator was applied to the entire operation.

The following is an example demonstrating inverted return values.

ArrayList<Value> lowTuple = new ArrayList<Value>();
lowTuple.add(Value.get(1400));
lowTuple.add(Value.NULL);

ArrayList<Value> highTuple = new ArrayList<Value>();
highTuple.add(Value.get(3500));
highTuple.add(Value.NULL);

Record between1400and3500 = client.operate(client.writePolicyDefault, whaleKey, 
    ListOperation.getByValueRange(listWhaleBinName, Value.get(lowTuple), Value.get(highTuple), 
        ListReturnType.COUNT | ListReturnType.INVERTED),
    ListOperation.getByValueRange(listWhaleBinName, Value.get(lowTuple), Value.get(highTuple), 
        ListReturnType.INDEX | ListReturnType.INVERTED),    
    ListOperation.getByValueRange(listWhaleBinName, Value.get(lowTuple), Value.get(highTuple), 
        ListReturnType.NONE | ListReturnType.INVERTED),   
    ListOperation.getByValueRange(listWhaleBinName, Value.get(lowTuple), Value.get(highTuple), 
        ListReturnType.RANK | ListReturnType.INVERTED),    
    ListOperation.getByValueRange(listWhaleBinName, Value.get(lowTuple), Value.get(highTuple), 
        ListReturnType.REVERSE_INDEX | ListReturnType.INVERTED),    
    ListOperation.getByValueRange(listWhaleBinName, Value.get(lowTuple), Value.get(highTuple), 
        ListReturnType.REVERSE_RANK | ListReturnType.INVERTED),    
    ListOperation.getByValueRange(listWhaleBinName, Value.get(lowTuple), Value.get(highTuple), 
        ListReturnType.VALUE | ListReturnType.INVERTED)   
    );

List<?> returnWhaleRange = between1400and3500.getList(listWhaleBinName);

System.out.println("The current whale migration list is: " + postCreate.getValue(listWhaleBinName) + "\n");
System.out.println("For the whales who migrate between 1400 and 3500 miles...");
System.out.println("Return INVERTED COUNT: " + returnWhaleRange.get(0));
System.out.println("Return INVERTED INDEX: " + returnWhaleRange.get(1));
System.out.println("Return INVERTED NONE: has no return value.");
System.out.println("Return INVERTED RANK: " + returnWhaleRange.get(2));
System.out.println("Return INVERTED REVERSE_INDEX: " + returnWhaleRange.get(3));
System.out.println("Return INVERTED REVERSE_RANK: " + returnWhaleRange.get(4));
System.out.println("Return INVERTED Values: " + returnWhaleRange.get(5));

Output:

The current whale migration list is: [[1420, beluga whale, Beaufort Sea, Bering Sea], [13988, gray whale, Baja California, Chukchi Sea], [1278, north pacific right whale, Japan, Sea of Okhotsk], [5100, humpback whale, Columbia, Antarctic Peninsula], [3100, southern hemisphere blue whale, Corcovado Gulf, The Galapagos], [1449, sei whale, Greenland, Gulf of Maine]]

For the whales who migrate between 1400 and 3500 miles...
Return INVERTED COUNT: 3
Return INVERTED INDEX: [1, 2, 3]
Return INVERTED NONE: has no return value.
Return INVERTED RANK: [0, 4, 5]
Return INVERTED REVERSE_INDEX: [4, 3, 2]
Return INVERTED REVERSE_RANK: [5, 0, 1]
Return INVERTED Values: [[13988, gray whale, Baja California, Chukchi Sea], [1278, north pacific right whale, Japan, Sea of Okhotsk], [5100, humpback whale, Columbia, Antarctic Peninsula]]

Highlighting how policies shape application transactions

Each data type operation has a write policy which can be set per CDT write/put operation to optionally:

Just-in-time sort the data being operated on.
Apply flags that instruct Aerospike’s transaction write behavior.

Create and set a MapPolicy or ListPolicy with the proper sort and write flags to change how Aerospike processes a transaction.

MapOrder and ListOrder, Just-in-time Sorting for an Operation

By default, Maps and Lists are stored unordered. There are explicit techniques to store a list or map in order. The Map data in this notebook is key sorted. Please refer to the code snippet creating the map data (above) for an example of this. There are examples of ordering lists in the notebook Modeling Using Lists.

Applying a MapOrder or ListOrder has performance implications on operation performance. This can be a reason to apply a MapOrder or ListOrder when working with data. To understand the relative worst-case time complexity of Aerospike operations go here for lists and here for maps.

Whether to allow duplicates in a list is a function of ListOrder.

Note: Aerospike finds that worst-case performance can be helpful in determining how to prioritize application use-cases against one another, but do not set realistic performance expectations for Aerospike Database. An example where they help is asking tough questions, like, “the worst case time complexity for operation A is X, is operation A important enough to do daily or just monthly in light of the other workloads that are more time sensitive?”

Write Flags

The following are lists of write flags for Lists and Maps. Beneath each are example transactions.

A powerful use case for Aerospike is to group operations together into single-record atomic transactions using the Operate method. This technique is used above in this notebook. When applying transactions to data, there are common circumstances where:

All possible operations should be executed in a fault tolerant manner
Specific operation failure should cause all operations to fail

Write flags can be used in any combination, as appropriate to the application and Aerospike operation being applied.

Write Flags for all CDTs

DEFAULT
- For Lists, allow duplicate values and insertions at any index.
- For Maps, allow map create or updates.
NO_FAIL: Do not raise an error if a CDT item is denied due to write flag constraints.
PARTIAL: Allow other valid CDT items to be committed if a CDT item is denied due to write flag constraints.

These flags provide fault tolerance to transactions. Apply some combination of the above three flags–DEFAULT, NO_FAIL, and PARTIAL–to operations by using “logical or” as demonstrated below. All other write flags set conditions for operations.

Note: Without NO_FAIL, operations that fail due to the below policies will throw either error code 24 or 26.

Default Examples

All of the above code snippets use a Default write flag policy. These operations are unrestricted by write policies.

No Fail Examples

All of the examples in the following sections show both an exception caused by a write flag, and then pair the demonstrated write flag with No Fail to show how the same operation can fail silently.

Partial Flag Example

Partial is generally used only in a transaction containing operations using the No Fail write flag. Otherwise, the transaction would contain no failures to overlook. The following example are a list and map transaction combining both failing and successful map and list operations.

// create policy to apply and data to trigger operation failure
Integer inBoundsIndex = 0;
Integer outOfBoundsIndex = 20;

HashMap <Value, Value> mapCoords4 = new HashMap <Value, Value>();
mapCoords4.put(Value.get("lat"), Value.get(0));
mapCoords4.put(Value.get("long"), Value.get(0));

Integer existingObsKey = 13456;

Integer listPartialWriteFlags = ListWriteFlags.INSERT_BOUNDED 
                                | ListWriteFlags.NO_FAIL 
                                | ListWriteFlags.PARTIAL;
ListPolicy listPartialWritePolicy = new ListPolicy(ListOrder.UNORDERED, listPartialWriteFlags);

Integer mapPartialWriteFlags = MapWriteFlags.CREATE_ONLY 
                                | MapWriteFlags.NO_FAIL 
                                | MapWriteFlags.PARTIAL;
MapPolicy mapPartialWritePolicy = new MapPolicy(MapOrder.KEY_ORDERED, mapPartialWriteFlags);


// create fresh record
Integer partialFlagKeyName = 6;
Key partialFlagKey = new Key(nestedCDTNamespaceName, nestedCDTSetName, partialFlagKeyName);

Bin bin1 = new Bin(listWhaleBinName, whaleMigration);
Record putDataIn = client.operate(null, partialFlagKey,
        Operation.put(bin1),
        MapOperation.putItems(mapObsPolicy, mapObsBinName, mapObs)
    );
Record partialDataPutIn = client.get(client.writePolicyDefault, partialFlagKey);


// one failed and one successful operation for both list and map
Record partialSuccessOp = client.operate(null, partialFlagKey,
        ListOperation.insert(listPartialWritePolicy, listWhaleBinName, outOfBoundsIndex, Value.get(newWhaleMigration)),
        ListOperation.set(listPartialWritePolicy, listWhaleBinName, inBoundsIndex, Value.get(newWhaleMigration)),        
        MapOperation.put(mapPartialWritePolicy, mapObsBinName, Value.get(existingObsKey), Value.get(mapCoords4)),
        MapOperation.put(mapPartialWritePolicy, mapObsBinName, Value.get(newObsKey), Value.get(mapCoords3))
    );
Record partialSuccessData = client.get(client.writePolicyDefault, partialFlagKey);
System.out.println ("Failed to add a 5th item.\nSucceeded at changing the first item.\n");
System.out.println ("Original List: " + partialDataPutIn.getValue(listWhaleBinName) + "\n");
System.out.println ("Updated List: " + partialSuccessData.getValue(listWhaleBinName) + "\n\n"); 
System.out.println ("Failed to modify an exiting observation.\nSucceeded at adding a new observation.\n");
System.out.println ("Original Map: " + partialDataPutIn.getValue(mapObsBinName) + "\n");
System.out.println ("Updated Map: " + partialSuccessData.getValue(mapObsBinName) + "\n\nFor more about the failed operations, see the examples below.");

Boolean partialExampleRecordDeleted=client.delete(null, partialFlagKey);

Output:

Failed to add a 5th item.
Succeeded at changing the first item.

Original List: [[1420, beluga whale, Beaufort Sea, Bering Sea], [13988, gray whale, Baja California, Chukchi Sea], [1278, north pacific right whale, Japan, Sea of Okhotsk], [5100, humpback whale, Columbia, Antarctic Peninsula], [3100, southern hemisphere blue whale, Corcovado Gulf, The Galapagos]]

Updated List: [[1449, sei whale, Greenland, Gulf of Maine], [13988, gray whale, Baja California, Chukchi Sea], [1278, north pacific right whale, Japan, Sea of Okhotsk], [5100, humpback whale, Columbia, Antarctic Peninsula], [3100, southern hemisphere blue whale, Corcovado Gulf, The Galapagos]]

Failed to modify an exiting observation.
Succeeded at adding a new observation.

Original Map: {12345={lat=-85, long=-130}, 13456={lat=-25, long=-50}, 14567={lat=35, long=30}}

Updated Map: {12345={lat=-85, long=-130}, 13456={lat=-25, long=-50}, 14567={lat=35, long=30}, 15678={lat=95, long=110}}

For more about the failed operations, see the examples below.

Write Flags for Lists Only:

INSERT_BOUNDED: Enforce list boundaries when inserting. Do not allow values to be inserted at index outside current list boundaries.
ADD_UNIQUE: Only add unique values.

Insert Bounded Example

// create policy to apply and data to break policy
Integer outOfBoundsIndex = 20;

ListPolicy listInsertBoundedPolicy = new ListPolicy(ListOrder.UNORDERED, ListWriteFlags.INSERT_BOUNDED);
ListPolicy listBoundedNoFailPolicy = new ListPolicy(ListOrder.UNORDERED, ListWriteFlags.INSERT_BOUNDED 
                                                                            | ListWriteFlags.NO_FAIL);

// create fresh record
Integer whaleBoundedKeyName = 7;

Bin bin1 = new Bin(listWhaleBinName, whaleMigration);

Key whaleBoundedKey = new Key(nestedCDTNamespaceName, nestedCDTSetName, whaleBoundedKeyName);

client.put(client.writePolicyDefault, whaleBoundedKey, bin1);
Record ibDataPutIn = client.get(null, whaleBoundedKey);
System.out.println("Data in the record: " + ibDataPutIn.getValue(listWhaleBinName) + "\n");


// fail for INSERT_BOUNDED
try {
    Record ibFail = client.operate(client.writePolicyDefault, whaleBoundedKey,
        ListOperation.insert(listInsertBoundedPolicy, listWhaleBinName, outOfBoundsIndex, Value.get(newWhaleMigration))
    );
    System.out.println("The code does not get here.");
} 
catch(Exception e) {
    System.out.println("Out of Bounds Attempt 1: Exception caught.");
    Record ibNoFail = client.operate(client.writePolicyDefault, whaleBoundedKey,
        ListOperation.insert(listBoundedNoFailPolicy, listWhaleBinName, outOfBoundsIndex, Value.get(newWhaleMigration))
    );
    Record ibNoFailData = client.get(client.writePolicyDefault, whaleBoundedKey);
    if(ibNoFailData.getValue(listWhaleBinName).equals(ibDataPutIn.getValue(listWhaleBinName))) {
        System.out.println("Out of Bounds Attempt 2: No operation was executed. Error was suppressed by NO_FAIL.\n");
    }
}

Record noIB = client.operate(client.writePolicyDefault, whaleBoundedKey,
        ListOperation.insert(listWhaleBinName, outOfBoundsIndex, Value.get(newWhaleMigration))
);
Record noIBData = client.get(null, whaleBoundedKey);
System.out.println("Without Insert Bounded, a series of nulls is inside the Bin: " + noIBData.getValue(listWhaleBinName));

Output:

Data in the record: [[1420, beluga whale, Beaufort Sea, Bering Sea], [13988, gray whale, Baja California, Chukchi Sea], [1278, north pacific right whale, Japan, Sea of Okhotsk], [5100, humpback whale, Columbia, Antarctic Peninsula], [3100, southern hemisphere blue whale, Corcovado Gulf, The Galapagos]]

Out of Bounds Attempt 1: Exception caught.
Out of Bounds Attempt 2: No operation was executed. Error was suppressed by NO_FAIL.

Without Insert Bounded, a series of nulls is inside the Bin: [[1420, beluga whale, Beaufort Sea, Bering Sea], [13988, gray whale, Baja California, Chukchi Sea], [1278, north pacific right whale, Japan, Sea of Okhotsk], [5100, humpback whale, Columbia, Antarctic Peninsula], [3100, southern hemisphere blue whale, Corcovado Gulf, The Galapagos], null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, [1449, sei whale, Greenland, Gulf of Maine]]

Add Unique Example

// create policy to apply
ListPolicy listAddUniquePolicy = new ListPolicy(ListOrder.UNORDERED, ListWriteFlags.ADD_UNIQUE);
ListPolicy listAddUniqueNoFailPolicy = new ListPolicy(ListOrder.UNORDERED, ListWriteFlags.ADD_UNIQUE 
                                                                            | ListWriteFlags.NO_FAIL);

// create fresh record
Integer whaleAddUniqueKeyName = 8;
Bin bin1 = new Bin(listWhaleBinName, whaleMigration);
Key whaleAddUniqueKey = new Key(nestedCDTNamespaceName, nestedCDTSetName, whaleAddUniqueKeyName);
client.put(client.writePolicyDefault, whaleAddUniqueKey, bin1);
Record auDataPutIn = client.get(null, whaleAddUniqueKey);


// successful ADD_UNIQUE operation
Record auSuccess = client.operate(client.writePolicyDefault, whaleAddUniqueKey,
    ListOperation.append(listAddUniquePolicy, listWhaleBinName, Value.get(newWhaleMigration))
);
Record auSuccessData = client.get(null, whaleAddUniqueKey);

System.out.println("Data after the unique add of " + newWhaleMigration + ": " + auSuccessData.getValue(listWhaleBinName) + "\n");


// fail for 2nd ADD_UNIQUE
try {
    Record auFail = client.operate(client.writePolicyDefault, whaleAddUniqueKey,
        ListOperation.append(listAddUniquePolicy, listWhaleBinName, Value.get(newWhaleMigration))
    );
    System.out.println("The code does not get here.");
}
catch(Exception e) {
    System.out.println("Non-Unique Add 1: Exception caught.");
    Record auNoFail = client.operate(client.writePolicyDefault, whaleAddUniqueKey,
        ListOperation.append(listAddUniqueNoFailPolicy, listWhaleBinName, Value.get(newWhaleMigration))
    );
    Record auNoFailData = client.get(null, whaleAddUniqueKey);
    if(auNoFailData.getValue(listWhaleBinName).equals(auSuccessData.getValue(listWhaleBinName))) {
        System.out.println("Non-Unique Add 2: No operation was executed. Error was suppressed by NO_FAIL.\n");
    }
}

Record noAU = client.operate(client.writePolicyDefault, whaleAddUniqueKey,
        ListOperation.append(listWhaleBinName, Value.get(newWhaleMigration))
);
Record noAUData = client.get(null, whaleAddUniqueKey);
System.out.println("Without Add Unique here, the tuple for a sei whale is there 2x: " + noAUData.getValue(listWhaleBinName));

Output:

Data after the unique add of [1449, sei whale, Greenland, Gulf of Maine]: [[1420, beluga whale, Beaufort Sea, Bering Sea], [13988, gray whale, Baja California, Chukchi Sea], [1278, north pacific right whale, Japan, Sea of Okhotsk], [5100, humpback whale, Columbia, Antarctic Peninsula], [3100, southern hemisphere blue whale, Corcovado Gulf, The Galapagos], [1449, sei whale, Greenland, Gulf of Maine]]

Non-Unique Add 1: Exception caught.
Non-Unique Add 2: No operation was executed. Error was suppressed by NO_FAIL.

Without Add Unique here, the tuple for a sei whale is there 2x: [[1420, beluga whale, Beaufort Sea, Bering Sea], [13988, gray whale, Baja California, Chukchi Sea], [1278, north pacific right whale, Japan, Sea of Okhotsk], [5100, humpback whale, Columbia, Antarctic Peninsula], [3100, southern hemisphere blue whale, Corcovado Gulf, The Galapagos], [1449, sei whale, Greenland, Gulf of Maine], [1449, sei whale, Greenland, Gulf of Maine]]

Write Flags for Maps Only:

CREATE_ONLY: If the key already exists, the item will be denied.
UPDATE_ONLY: If the key already exists, the item will be overwritten. If the key does not exist, the item will be denied.

Create Only Example

// create modify data and policy to apply
HashMap <Value, Value> mapCoords4 = new HashMap <Value, Value>();
mapCoords4.put(Value.get("lat"), Value.get(0));
mapCoords4.put(Value.get("long"), Value.get(0));

MapPolicy mapCreateOnlyPolicy = new MapPolicy(MapOrder.KEY_ORDERED, MapWriteFlags.CREATE_ONLY);
MapPolicy mapCreateOnlyNoFailPolicy = new MapPolicy(MapOrder.KEY_ORDERED, MapWriteFlags.CREATE_ONLY 
                                                                            | MapWriteFlags.NO_FAIL);

// create fresh record
Integer obsCreateOnlyKeyName = 9;
Key obsCreateOnlyKey = new Key(nestedCDTNamespaceName, nestedCDTSetName, obsCreateOnlyKeyName);
Record putDataIn = client.operate(client.writePolicyDefault, obsCreateOnlyKey,
    MapOperation.putItems(mapObsPolicy, mapObsBinName, mapObs)
    );
Record coDataPutIn = client.get(null, obsCreateOnlyKey);


// success for CREATE_ONLY
Record coSuccess = client.operate(client.writePolicyDefault, obsCreateOnlyKey,
    MapOperation.put(mapCreateOnlyPolicy, mapObsBinName, Value.get(newObsKey), Value.get(mapCoords3))
);
Record coSuccessData = client.get(null, obsCreateOnlyKey);
System.out.println("Created record and new key " + newObsKey + ". The data is now: " + coSuccessData.getValue(mapObsBinName) + "\n");


// fail for CREATE_ONLY
try {
    Record coFail = client.operate(client.writePolicyDefault, obsCreateOnlyKey,
        MapOperation.put(mapCreateOnlyPolicy, mapObsBinName, Value.get(newObsKey), Value.get(mapCoords4))
    );
    System.out.println("The code does not get here.");
}
catch(Exception e) {
    System.out.println("Update attempt 1: Exception caught.");
    Record coNoFail = client.operate(client.writePolicyDefault, obsCreateOnlyKey,
        MapOperation.put(mapCreateOnlyNoFailPolicy, mapObsBinName, Value.get(newObsKey), Value.get(mapCoords4))
    );
    Record coNoFailData = client.get(null, obsCreateOnlyKey);
    if(coNoFailData.getValue(mapObsBinName).equals(coSuccessData.getValue(mapObsBinName))) {
        System.out.println("Update attempt 2: No operation was executed. Error was suppressed by NO_FAIL.\n");
    }
}

Record noCO = client.operate(client.writePolicyDefault, obsCreateOnlyKey, 
    MapOperation.put(mapObsPolicy, mapObsBinName, Value.get(newObsKey), Value.get(mapCoords4))
);
Record noCOData = client.get(null, obsCreateOnlyKey);
System.out.println("Without Create Only, the observation at 15678 is overwritten: " + noCOData.getValue(mapObsBinName));

Boolean createOnlyExampleRecordDeleted=client.delete(null, obsCreateOnlyKey);

Output:

Created record and new key 15678. The data is now: {12345={lat=-85, long=-130}, 13456={lat=-25, long=-50}, 14567={lat=35, long=30}, 15678={lat=95, long=110}}

Update attempt 1: Exception caught.
Update attempt 2: No operation was executed. Error was suppressed by NO_FAIL.

Without Create Only, the observation at 15678 is overwritten: {12345={lat=-85, long=-130}, 13456={lat=-25, long=-50}, 14567={lat=35, long=30}, 15678={lat=0, long=0}}

Update Only Example

// create policy to apply
MapPolicy mapUpdateOnlyPolicy = new MapPolicy(MapOrder.KEY_ORDERED, MapWriteFlags.UPDATE_ONLY);
MapPolicy mapUpdateOnlyNoFailPolicy = new MapPolicy(MapOrder.KEY_ORDERED, MapWriteFlags.UPDATE_ONLY 
                                                                            | MapWriteFlags.NO_FAIL);

// create Aerospike data elements for a fresh record
Integer obsUpdateOnlyKeyName = 10;
Key obsUpdateOnlyKey = new Key(nestedCDTNamespaceName, nestedCDTSetName, obsUpdateOnlyKeyName);

Record uoPutDataIn = client.operate(client.writePolicyDefault, obsUpdateOnlyKey,
    MapOperation.putItems(mapObsPolicy, mapObsBinName, mapObs)
    );
Record uoDataPutIn = client.get(null, obsUpdateOnlyKey);
System.out.println("Created record: " + uoDataPutIn.getValue(mapObsBinName) + "\n");


// fail for UPDATE_ONLY
try {
    Record uoFail = client.operate(client.writePolicyDefault, obsUpdateOnlyKey,
        MapOperation.put(mapUpdateOnlyPolicy, mapObsBinName, Value.get(newObsKey), Value.get(mapCoords3))
    );
    System.out.println("The code does not get here.");
}
catch(Exception e) {
    System.out.println("Create Attempt 1: Exception caught.");
    Record uoNoFail = client.operate(client.writePolicyDefault, obsUpdateOnlyKey,
        MapOperation.put(mapUpdateOnlyNoFailPolicy, mapObsBinName, Value.get(newObsKey), Value.get(mapCoords3))
    );
    Record uoNoFailData = client.get(null, obsUpdateOnlyKey);
    if(uoNoFailData.getValue(mapObsBinName).equals(uoDataPutIn.getValue(mapObsBinName))){
        System.out.println("Create Attempt 2: No operation was executed. Error was suppressed by NO_FAIL.\n");
    }
}

Record noUO = client.operate(client.writePolicyDefault, obsUpdateOnlyKey, 
        MapOperation.put(mapObsPolicy, mapObsBinName, Value.get(newObsKey), Value.get(mapCoords3))
);
Record noUOData = client.get(null, obsUpdateOnlyKey);

// success for UPDATE_ONLY
Record uoSuccess = client.operate(client.writePolicyDefault, obsUpdateOnlyKey,
    MapOperation.put(mapUpdateOnlyPolicy, mapObsBinName, Value.get(existingObsKey), Value.get(mapCoords4))
);
Record uoSuccessData = client.get(null, obsUpdateOnlyKey);
System.out.println("Using update only, the value of an existing key " + existingObsKey + " can be updated: " + uoSuccessData.getValue(mapObsBinName) + "\n");

Boolean uoExampleRecordDeleted=client.delete(null, obsUpdateOnlyKey);

Output:

Created record: {12345={lat=-85, long=-130}, 13456={lat=-25, long=-50}, 14567={lat=35, long=30}}

Create Attempt 1: Exception caught.
Create Attempt 2: No operation was executed. Error was suppressed by NO_FAIL.

Using update only, the value of an existing key 13456 can be updated: {12345={lat=-85, long=-130}, 13456={lat=0, long=0}, 14567={lat=35, long=30}, 15678={lat=95, long=110}}

Notebook Cleanup

Truncate the Set

Truncate the set from the Aerospike Database.

import com.aerospike.client.policy.InfoPolicy;
InfoPolicy infoPolicy = new InfoPolicy();

client.truncate(infoPolicy, nestedCDTNamespaceName, nestedCDTSetName, null);
System.out.println("Set Truncated.");

Output:
Set Truncated.

Close the Client connections to Aerospike

client.close();
System.out.println("Server connection(s) closed.");

Output:

Server connection(s) closed.

Takeaways – CDTs Provide Flexible Document-Oriented Data Power

Aerospike Collection Data Types...

facilitate complex data structures by supporting nesting through the use of contexts (CTXs)
provide intuitive and flexible return types options from operations
support policies that empower efficient and flexible transaction processing

What's Next?

Next Steps

Have questions? Don't hesitate to reach out if you have additional questions about data modeling at https://discuss.aerospike.com/c/how-developers-are-using-aerospike/data-modeling/143.

Want to check out other Java notebooks?

Are you running this from Binder? Download the Aerospike Notebook Repo and work with Aerospike Database and Jupyter locally using a Docker container.

Additional Resources

Want to get started with Java? Download or install the Aerospike Java Client.
(https://aerospike.com/apidocs/java/com/aerospike/client/cdt/MapOperation.html).
What are Namespaces, Sets, and Bins? Check out the Aerospike Data Model.
How robust is the Aerospike Database? Browses the Aerospike Database Architecture.

Advanced Collection Data Types

For an interactive Jupyter notebook experience:

Prerequisites​

Notebook Setup​

Import Jupyter Java Integration​

Start Aerospike​

Download the Aerospike Java Client​

Start the Aerospike Java Client and Connect​

Create CDT Data, Put into Aerospike, and Print It​

Using Contexts (CTXs) to work with Nested CDTs​

What is a Nested CDT?​

What is a Context?​

Look up a Map CTX in a Bin by Mapkey​

Drill down into a List or Map​

Create a CTX Example​

Choosing the Return Type Options for CDTs​

Standard Return Type Options for CDTs​

Additional Return Type Options for Maps​

Invert the Operation Results for CDT Operations​

Highlighting how policies shape application transactions​

MapOrder and ListOrder, Just-in-time Sorting for an Operation​

Write Flags​

Write Flags for all CDTs​

Default Examples​

No Fail Examples​

Partial Flag Example​

Write Flags for Lists Only:​

Insert Bounded Example​

Add Unique Example​

Write Flags for Maps Only:​

Create Only Example​

Update Only Example​

Notebook Cleanup​

Truncate the Set​

Close the Client connections to Aerospike​

Takeaways – CDTs Provide Flexible Document-Oriented Data Power​

What's Next?​

Next Steps​

Additional Resources​

Prerequisites

Notebook Setup

Import Jupyter Java Integration

Start Aerospike

Download the Aerospike Java Client

Start the Aerospike Java Client and Connect

Create CDT Data, Put into Aerospike, and Print It

Using Contexts (CTXs) to work with Nested CDTs

What is a Nested CDT?

What is a Context?

Look up a Map CTX in a Bin by Mapkey

Drill down into a List or Map

Create a CTX Example

Choosing the Return Type Options for CDTs

Standard Return Type Options for CDTs

Additional Return Type Options for Maps

Invert the Operation Results for CDT Operations

Highlighting how policies shape application transactions

MapOrder and ListOrder, Just-in-time Sorting for an Operation

Write Flags

Write Flags for all CDTs

Default Examples

No Fail Examples

Partial Flag Example

Write Flags for Lists Only:

Insert Bounded Example

Add Unique Example

Write Flags for Maps Only:

Create Only Example

Update Only Example

Notebook Cleanup

Truncate the Set

Close the Client connections to Aerospike

Takeaways – CDTs Provide Flexible Document-Oriented Data Power

What's Next?

Next Steps

Additional Resources