adaptivemd.mongodb package

Submodules

adaptivemd.mongodb.base module

class adaptivemd.mongodb.base.StorableMixin

Bases: object

Mixin that allows objects of the class to to be stored using netCDF+

INSTANCE_UUID = [1806962750L, 11303L, 4583L, 143L, 11L]

CREATION_COUNT = 0L

ACTIVE_LONG = 143162338427886574775798134289748459528L

static get_uuid()

Create a new unique ID :returns: the unique number for an object in the project :rtype: long

named(name)

Attach a .name property to an object

Parameters:name (str) – the name of the object Returns:the object itself for chaining Return type:self

idx(store)

Return the index which is used for the object in the given store.

Once you store a storable object in a store it gets assigned a unique number that can be used to retrieve the object back from the store. This function will ask the given store if the object is stored if so what the used index is.

Parameters:store (ObjectStore) – the store in which to ask for the index Returns:the integer index for the object of it exists or None else Return type:int or None

cls

Return the class name as a string

Returns:the class name Return type:str

classmethod base()

Return the most parent class actually derived from StorableMixin

Important to determine which store should be used for storage

Returns:the base class Return type:type

base_cls_name

Return the name of the base class

Returns:the string representation of the base class Return type:str

base_cls

Return the base class

Returns:the base class Return type:type

See also

base()

classmethod descendants()

Return a list of all subclassed objects

Returns:list of subclasses of a storable object Return type:list of type

static objects()

Returns a dictionary of all storable objects

Returns:dict of str – a dictionary of all subclassed objects from StorableMixin. The name points to the class Return type:type

classmethod args()

Return a list of args of the __init__ function of a class

Returns:the list of argument names. No information about defaults is included. Return type:list of str

to_dict()

Convert object into a dictionary representation

Used to convert the dictionary into JSON string for serialization

Returns:the dictionary representing the (immutable) state of the object Return type:dict

classmethod from_dict(dct)

Reconstruct an object from a dictionary representation

Parameters:dct (dict) – the dictionary containing a state representation of the class. Returns:the reconstructed storable object Return type:StorableMixin

adaptivemd.mongodb.base.create_to_dict(keys_to_store)

Create a to_dict function from a list of attributes

Parameters:keys_to_store (list of str) – the attributes used in { attr: getattr(self, attr) } Returns:the to_dict function Return type:function

adaptivemd.mongodb.cache module

class adaptivemd.mongodb.cache.Cache

Bases: object

A cache like dict

count

int – the number of strong references int : the number of weak references

size

int – the maximal number of strong references, -1 if infinite int : the maximal number of weak references, -1 if infinite

get(item, default=None)

get value by key if it exists, None else

Parameters:

• item (object) – key to select element in cache
• default (object) – return value if item is not present in cache

Returns:

cached value at key item if present, returns default otherwise

Return type:

object or None

transfer(old_cache)

Transfer values between caches

Useful if during run-time a cache is replaced by another instance

Parameters:old_cache (the cache from which this cache is to be filled) –

get_silent(item, default=None)

get value by key if it exists, None else

Parameters:

• item (object) – key to select element in cache
• default (object) – return value if item is not present in cache

Returns:

cached value at key item if present, returns default otherwise

Return type:

object or None

class adaptivemd.mongodb.cache.NoCache

Bases: adaptivemd.mongodb.cache.Cache

A virtual cache the contains no elements

count

size

items()

transfer(old_cache)

clear()

class adaptivemd.mongodb.cache.MaxCache

Bases: dict, adaptivemd.mongodb.cache.Cache

A dictionary, can hold infinite strong references

count

size

class adaptivemd.mongodb.cache.LRUCache(size_limit)

Bases: adaptivemd.mongodb.cache.Cache

Implements a simple Least Recently Used Cache

Very simple using collections.OrderedDict. The size can be changed during run-time

count

size

size_limit

clear()

class adaptivemd.mongodb.cache.WeakLRUCache(size_limit=100, weak_type='value')

Bases: adaptivemd.mongodb.cache.Cache

Implements a cache that keeps weak references to all elements

In addition it uses a simple Least Recently Used Cache to make sure a portion of the last used elements are still present. Usually this number is 100.

Parameters:size_limit (int) – integer that defines the size of the LRU cache. Default is 100.

__init__(size_limit=100, weak_type='value')

Parameters:size_limit (int) – integer that defines the size of the LRU cache. Default is 100.

count

size

clear()

size_limit

get_silent(item)

Return item from the without reordering the LRU

Parameters:item (object) – the item index to be retrieved from the cache Returns:the requested object if it exists else None Return type:object or None

keys()

values()

class adaptivemd.mongodb.cache.WeakValueCache(*args, **kwargs)

Bases: weakref.WeakValueDictionary, adaptivemd.mongodb.cache.Cache

Implements a cache that keeps weak references to all elements

count

size

class adaptivemd.mongodb.cache.WeakKeyCache(dict=None)

Bases: weakref.WeakKeyDictionary, adaptivemd.mongodb.cache.Cache

Implements a cache that keeps weak references to all elements

count

size

adaptivemd.mongodb.dictify module

class adaptivemd.mongodb.dictify.ObjectJSON(unit_system=None)

Bases: object

A simple implementation of a pickle algorithm to create object that can be converted to json and back

allow_marshal = True

prevent_unsafe_modules = False

allowed_storable_atomic_types = [<type 'int'>, <type 'float'>, <type 'bool'>, <type 'long'>, <type 'str'>, <Mock id='140616587076880'>, <Mock id='140616587077008'>, <Mock id='140616587077136'>, <Mock id='140616587077264'>, <Mock id='140616587077392'>, <Mock id='140616587077520'>, <Mock id='140616587077648'>, <Mock id='140616587077776'>, <Mock id='140616587077904'>, <Mock id='140616587078032'>]

safe_modules = ['numpy', 'math', 'pandas', 'mdtraj', 'simtk', 'simtk.unit', 'simtk.openmm']

update_class_list()

simplify_object(obj)

simplify(obj, base_type='')

build(obj)

static unit_to_symbol(unit)

static callable_to_dict(c)

Turn a callable function of class into a dictionary

Used for conversion to JSON

Parameters:c (callable (function or class with __call__)) – the function to be turned into a dict representation Returns:the dict representation of the callable Return type:dict

static callable_from_dict(c_dict)

Turn a dictionary back in a callable function or class

Used for conversion from JSON

Parameters:c_dict (dict) – the dictionary that contains the information Returns:the reconstructed callable function or class Return type:callable

to_json(obj, base_type='')

to_json_object(obj)

from_json(json_string)

from_simple_dict(simplified)

to_simple_dict(obj, base_type='')

class adaptivemd.mongodb.dictify.UUIDObjectJSON(storage, unit_system=None)

Bases: adaptivemd.mongodb.dictify.ObjectJSON

simplify(obj, base_type='')

build(obj)

adaptivemd.mongodb.dictify.word_wrap(string, width=80)

adaptivemd.mongodb.file module

class adaptivemd.mongodb.file.FileStore(name, content_class)

Bases: adaptivemd.mongodb.object.ObjectStore

initialize()

restore()

consume_one(test_fnc=None)

modify_one(key, value, update)

modify_test_one(test_fnc, key, value, update)

load_indices()

cache_all()

find_one(dct)

load(idx)

Returns an object from the storage.

Parameters:idx (int) – the integer index of the object to be loaded Returns:the loaded object Return type:mongodb.base.StorableMixin

save(obj)

Saves an object to the storage.

Parameters:obj (mongodb.base.StorableMixin) – the object to be stored

class adaptivemd.mongodb.file.DataDict(data)

Bases: adaptivemd.mongodb.base.StorableMixin

Delegate to the contained .data object

adaptivemd.mongodb.mongodb module

class adaptivemd.mongodb.mongodb.MongoDBStorage(filename, mode=None)

Bases: object

Extension of the pymongo wrapper for easier storage of python objects

Create a mongodb storage for complex objects

Parameters:

• filename (string) – name of the mongodb database
• mode (str) – the mode of file creation, one of ‘w’ (write), ‘a’ (append) or ‘r’ (read-only) None, which will append any existing files (equal to append), is the default.

Notes

You can safely open a storage from multiple instances. These will cross update.

version

objects

Return a dictionary of all objects stored.

find_store(obj)

Return the default store used for an storable object

Parameters:obj (mongodb.StorableMixin) – the storable object to be tested Returns:the store that is used by default to store the given storable obj Return type:mongodb.ObjectStore

update_storable_classes()

Update the internal list of all objects that are subclassed from StorableMixin

If you create your own subclass of a storable object then you need to call this function to update the list so that you can load and save instances of your new class

__init__(filename, mode=None)

Create a mongodb storage for complex objects

Parameters:

• filename (string) – name of the mongodb database
• mode (str) – the mode of file creation, one of ‘w’ (write), ‘a’ (append) or ‘r’ (read-only) None, which will append any existing files (equal to append), is the default.

Notes

You can safely open a storage from multiple instances. These will cross update.

close()

Close the DB connection

classmethod list_storages()

classmethod delete_storage(name)

check_version()

write_meta()

create_store(store, register_attr=True)

Create a special variable type obj.name that can hold storable objects

Parameters:

• store (mongodb.ObjectStore) – the store to be added to this storage
• register_attr (bool) –

  if True the store will be added to the storage as an

  attribute with name name

finalize_stores()

Run initializations for all added stores.

This will make sure that all previously added stores are now usable. If you add more stores you need to call this again. The reason this is done at all is that stores might reference each other and so no unique order of creation can be found. Thus you first create stores with all their dependencies and then finalize all of them together.

register_store(store, register_attr=True)

Add a object store to the file

An object store is a special type of variable that allows to store python objects

Parameters:

• store (mongodb.ObjectStore) – instance of the object store
• register_attr (bool) – if set to false the store will not be accesible as an attribute. True is the default.

list_stores()

Return a list of registered stores

Returns:list of stores that can be accessed using storage.[store] Return type:list of str

list_storable_objects()

Return a list of storable object base classes

Returns:list of base classes that can be stored using storage.save(obj) Return type:list of type

save(obj)

Save a storable object into the correct Storage in the netCDF file

Parameters:obj (StorableMixin) – the object to store Returns:the class name of the BaseClass of the stored object, which is needed when loading the object to identify the correct storage Return type:str

load(uuid)

Load an object from the storage

Parameters:uuid (uuid.UUID) – the uuid to be loaded Returns:the object loaded from the storage Return type:mongodb.StorableMixin

Notes

this only works in storages with uuids otherwise load directly from the sub-stores

cache_image()

Return an dict containing information about all caches

Returns:a nested dict containing information about the number and types of cached objects Return type:dict

adaptivemd.mongodb.object module

class adaptivemd.mongodb.object.ObjectStore(name, content_class)

Bases: adaptivemd.mongodb.base.StorableMixin

Base Class for storing complex objects in a netCDF4 file. It holds a reference to the store file.`

Variables:

• content_class (mongodb.base.StorableMixin) – a reference to the class type to be stored using this Storage. Must be subclassed from mongodb.base.StorableMixin
• cache (mongodb.cache.Cache) – a dictionary that holds references to all stored elements by index or string for named objects. This is only used for cached access if caching is not False. Must be of type mongodb.base.StorableMixin or subclassed.

Parameters:

• name (str) – the name of the store
• content_class (class) – the base class of the content, must be subclassed from StorableMixin

allowed_types = ['int', 'float', 'long', 'str', 'bool', 'numpy.float32', 'numpy.float64', 'numpy.int8', 'numpy.inf16', 'numpy.int32', 'numpy.int64', 'numpy.uint8', 'numpy.uinf16', 'numpy.uint32', 'numpy.uint64', 'index', 'length', 'uuid']

default_store_chunk_size = 256

default_cache = 10000

__init__(name, content_class)

Parameters:

• name (str) – the name of the store
• content_class (class) – the base class of the content, must be subclassed from StorableMixin

is_created()

to_dict()

check_size()

Perform an update in case the DB has been extended by an external source

Returns:returns True if an update was performed Return type:bool

register(storage)

Associate the object store to a specific storage with a given name

Parameters:storage (mongodb.NetCDFPlus) – the storage to be associated with

static create_uuid_index()

restore()

load_indices()

storage

Return the associated storage object

Returns:the referenced storage object Return type:mongodb.NetCDFPlus

simplifier

Return the simplifier instance used to create JSON serialization

Returns:the simplifier object used in the associated storage Return type:mongodb.dictify.StorableObjectJSON

set_caching(caching)

Set the caching mode for this store

Parameters:caching (mongodb.Cache) –

idx(obj)

Return the index in this store for a given object

Parameters:obj (mongodb.base.StorableMixin) – the object that can be stored in this store for which its index is to be returned Returns:The integer index of the given object or None if it is not stored yet Return type:int or None

proxy(item)

Return a proxy of a object for this store

Parameters:item (mongodb.base.StorableMixin) – or int The item or index that points to an object in this store and to which a proxy is requested.

get(item)

consume_one(test_fnc=None)

Remove one object and return it in the process

Parameters:test_fnc (function) – only objects that match by this function are considered Returns:if None then no object was altered, otherwise the changed object is returned Return type:None or StorableMixin

modify_one(key, value, update)

Change an attribute of one object

Parameters:

• key (str) – the attributes name to be changed
• value (object) – the old value to be found and changed
• update (object) – the new value to the changed into

Returns:

if None then no object was altered, otherwise the changed object is returned

Return type:

None or StorableMixin

modify_test_one(test_fnc, key, value, update)

Change an attribute of one object that matches a function

Parameters:

• test_fnc (function) – only objects that match by this function are considered
• key (str) – the attributes name to be changed
• value (object) – the old value to be found and changed
• update (object) – the new value to the changed into

Returns:

if None then no object was altered, otherwise the changed object is returned

Return type:

None or StorableMixin

clear_cache()

Clear the cache and force reloading

cache_all()

Load all samples as fast as possible into the cache

one

Returns one random object.

Returns:the content of the store Return type:StorableMixin

last

Returns the last saved object.

This is only accurate to seconds!

Returns:the content of the store Return type:StorableMixin

first

Returns the first saved object.

This is only accurate to seconds!

Returns:the content of the store Return type:StorableMixin

free()

Return the number of the next free index for this store

Returns:index – the number of the next free index in the storage. Used to store a new object. Return type:int

initialize()

Initialize the associated storage to allow for object storage. Mainly creates an index dimension with the name of the object.

find_one(dct)

load(idx)

Returns an object from the storage.

Parameters:idx (int) – the integer index of the object to be loaded Returns:the loaded object Return type:mongodb.base.StorableMixin

static reference(obj)

save(obj)

Saves an object to the storage.

Parameters:obj (mongodb.base.StorableMixin) – the object to be stored

add_single_to_cache(idx, json)

Add a single object to cache by json

Parameters:

• idx (int) – the index where the object was stored
• json (str) – json string the represents a serialized version of the stored object

adaptivemd.mongodb.proxy module

class adaptivemd.mongodb.proxy.LoaderProxy(store, idx)

Bases: object

A proxy that loads an underlying object if attributes are accessed

class adaptivemd.mongodb.proxy.DelayedLoader

Bases: object

Descriptor class to handle proxy objects in attributes

If a proxy is stored in an attribute then the full object will be returned

adaptivemd.mongodb.proxy.lazy_loading_attributes(*attributes)

Set attributes in the decorated class to be handled as lazy loaded objects.

An attribute that is added here will be turned into a special descriptor that will dynamically load an objects if it is represented internally as a LoaderProxy object and will return the real object, not the proxy!

The second thing you can do is that saving using the .write() command will automatically remove the real object and turn the stored object into a proxy

Notes

This decorator will obfuscate the __init__ signature in Python 2. This is fixed in Python 3.4+

adaptivemd.mongodb.syncvar module

class adaptivemd.mongodb.syncvar.SyncVariable(name, fix_fnc=None)

Bases: object

read(instance)

write(instance, v)

class adaptivemd.mongodb.syncvar.ObjectSyncVariable(name, store, fix_fnc=None)

Bases: adaptivemd.mongodb.syncvar.SyncVariable

class adaptivemd.mongodb.syncvar.JSONDataSyncVariable(name, fix_fnc=None)

Bases: adaptivemd.mongodb.syncvar.SyncVariable

adaptivemd.mongodb.version module

Module contents