Node storage

exception larch.nodestore.NodeCannotBeModified(node_id)

User called start_modification on node that cannot be modified.

exception larch.nodestore.NodeExists(node_id)

User tried to put a node that already exists in the store.

exception larch.nodestore.NodeMissing(node_store, node_id, error=None, error_msg='')

A node cannot be found from a NodeStore.

class larch.nodestore.NodeStore(allow_writes, node_size, codec)

Abstract base class for storing nodes externally.

The BTree class itself does not handle external storage of nodes. Instead, it is given an object that implements the API in this class. An actual implementation might keep nodes in memory, or store them on disk using a filesystem, or a database.

Node stores deal with nodes as byte strings: the codec encodes them before handing them to the store, and decodes them when it gets them from the store.

Each node has an identifier that is unique within the store. The identifier is an integer, and the caller makes the following guarantees about it:

  • it is a non-negative integer
  • new nodes are assigned the next consecutive one
  • it is never re-used

Further, the caller makes the following guarantees about the encoded nodes:

  • they have a strict upper size limit
  • the tree attempts to fill nodes as close to the limit as possible

The size limit is given to the node store at initialization time. It is accessible via the node_size property. Implementations of this API must handle that in some suitable way, preferably by inheriting from this class and calling its initializer.

self.max_value_size gives the maximum size of a value stored in a node.

A node store additionally stores some metadata, as key/value pairs, where both key and value is a shortish string. The whole pair must fit into a node, but more than one node can be used for metadata.

can_be_modified(node)

Can a node be modified?

commit()

Make sure all changes to are committed to the store.

Until this is called, there’s no guarantee that any of the changes since the previous commit are persistent.

get_metadata(key)

Return value that corresponds to a key.

get_metadata_keys()

Return list of all metadata keys.

get_node(node_id)

Return a node from the store.

Raise the NodeMissing exception if the node is not in the store (has never been, or has been removed). Raise other errors as suitable.

get_refcount(node_id)

Return the reference count for a node.

list_nodes()

Return list of ids of all nodes in store.

max_index_pairs()

Max number of index pairs in an index node.

put_node(node)

Put a new node into the store.

remove_metadata(key)

Remove a metadata key, and its corresponding value.

remove_node(node_id)

Remove a node from the store.

save_metadata()

Save metadata persistently, if applicable.

Not all node stores are persistent, and this method is not relevant to them. However, if the user does not call this method, none of the changes they make will be stored persistently even with a persistent store.

save_refcounts()

Save refcounts to disk.

This method only applies to node stores that persist.

set_metadata(key, value)

Set a metadata key/value pair.

set_refcount(node_id, refcount)

Set the reference count for a node.

start_modification(node)

Start modification of a node.

User must call this before modifying a node in place.

If a node cannot be modified, NodeCannotBeModified exception will be raised.

class larch.nodestore.NodeStoreTests

Re-useable tests for NodeStore implementations.

The NodeStore base class can’t be usefully instantiated itself. Instead you are supposed to sub-class it and implement the API in a suitable way for yourself.

This class implements a number of tests that the API implementation must pass. The implementation’s own test class should inherit from this class, and unittest.TestCase.

The test sub-class should define a setUp method that sets the following:

  • self.ns to an instance of the API implementation sub-class
  • self.node_size to the node size

Key size (self.key_bytes) is always 3.

assertEqualNodes(n1, n2)

Assert that two nodes are equal.

Equal means same keys, and same values for keys. Nodes can be either leaf or index ones.

key_bytes = 3
test_get_freezes_node()
test_has_no_node_zero_initially()
test_lists_no_nodes_initially()
test_lists_node_zero()
test_node_not_in_store_can_not_be_modified()
test_node_with_refcount_0_can_not_be_modified()
test_node_with_refcount_1_can_be_modified()
test_node_with_refcount_2_can_not_be_modified()
test_put_allows_to_overwrite_a_node()
test_put_allows_to_overwrite_a_node_after_upload_queue_push()
test_put_freezes_node()
test_puts_and_gets_same()
test_raises_error_when_getting_unknown_key()
test_raises_error_when_removing_unknown_key()
test_remove_raises_nodemissing_if_node_does_not_exist()
test_removes_metadata()
test_removes_node()
test_removes_node_from_upload_queue_if_one_exists()
test_returns_zero_count_for_unknown_node_id()
test_sets_existing_metadata()
test_sets_max_value_size()
test_sets_metadata()
test_sets_node_size()
test_sets_refcount()
test_sets_several_metadata_keys()
test_unfreezes_node_when_modification_starts()
test_updates_refcount()
exception larch.nodestore.NodeTooBig(node, node_size)

User tried to put a node that was too big into the store.

Previous topic

Node codecs

Next topic

Reference count storage

This Page