Salmon Run: A Generic BerkeleyDB store using DPL

I have written before about how much I liked the annotation driven persistence mechanism that BerkeleyDB Java Edition provides using its Direct Persistence Layer (DPL). I had an opportunity to look at it once more this weekend, this time with a view to persisting arbitary objects into Maps keyed by a unique String value.

The objects to be persisted are arbitary in the sense that the caller of the persistence code would know for sure what objects need to be persisted, and would persist the same class of objects into a given BerkeleyDB store. However, the code that did the persisting would not know what objects it was working with until it was instantiated by the caller. To do this, we define a generic StoreEntity object that persists objects of type V.

// StoreEntity.java
package com.mycompany.bdb;

import com.sleepycat.persist.model.Entity;
import com.sleepycat.persist.model.PrimaryKey;

@Entity
public class StoreEntity<V> {

  @PrimaryKey private String key;
  private V value;
  
  public StoreEntity() {
    super();
  }
  
  public String getKey() {
    return key;
  }
  
  public void setKey(String key) {
    this.key = key;
  }
  
  public V getValue() {
    return value;
  }
  
  public void setValue(V value) {
    this.value = value;
  }
}

The StoreEntity objects are persisted by a Store class which take care of initializing the database at startup in its init() method, and clean up resource handles in its destroy() method. It provides two methods getValue(String) to get an object of type V from the BerkeleyDB database and a setValue(String, V) to save the object V keyed by the String into the database.

// Store.java
package com.mycompany.bdb;

import java.io.File;

import org.apache.commons.io.FileUtils;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;

import com.sleepycat.je.Environment;
import com.sleepycat.je.EnvironmentConfig;
import com.sleepycat.persist.EntityStore;
import com.sleepycat.persist.PrimaryIndex;
import com.sleepycat.persist.StoreConfig;

public class Store<V> {

  private final Log log = LogFactory.getLog(getClass());
  
  private String dataDirectory;
  
  private Environment env;
  private EntityStore store;
  
  public void setDataDirectory(String dataDirectory) {
    this.dataDirectory = dataDirectory;
  }
  
  protected void init() throws Exception {
    File dataDir = new File(dataDirectory);
    if (! dataDir.exists()) {
      FileUtils.forceMkdir(dataDir);
    }
    EnvironmentConfig environmentConfig = new EnvironmentConfig();
    environmentConfig.setAllowCreate(true);
    environmentConfig.setTransactional(true);
    env = new Environment(dataDir, environmentConfig);
    StoreConfig storeConfig = new StoreConfig();
    storeConfig.setAllowCreate(true);
    storeConfig.setTransactional(true);
    store = new EntityStore(env, dataDir.getName(), storeConfig);
  }
  
  protected void destroy() throws Exception {
    if (store != null) {
      store.close();
    }
    if (env != null) {
      env.close();
    }
  }
  
  @SuppressWarnings("unchecked")
  public V getValue(String key) throws Exception {
    Class<?> entityClass = StoreEntity.class;
    PrimaryIndex<String,StoreEntity<V>> primaryIndex = 
      (PrimaryIndex<String,StoreEntity<V>>) store.getPrimaryIndex(
      key.getClass(), entityClass);
    StoreEntity<V> entity = (StoreEntity<V>) primaryIndex.get(key);
    return entity.getValue();
  }
  
  @SuppressWarnings("unchecked")
  public void setValue(String key, V value) throws Exception {
    StoreEntity<V> entity = new StoreEntity<V>();
    entity.setKey(key);
    entity.setValue(value);
    PrimaryIndex<String,StoreEntity<V>> primaryIndex = 
      (PrimaryIndex<String,StoreEntity<V>>) store.getPrimaryIndex(
      key.getClass(), entity.getClass());
    primaryIndex.put(entity);
  }
}

To use this, the client code looks something like this. Obviously, the client code would be better structured than this, probably pulling out the init() and destroy() calls out into its own init() and destroy() lifecycle methods, rather than lumping them together as shown below, but you get the idea.

public class ClientCode() {
  ...
  public void sampleCode() throws Exception {
    // initialize the store
    store = new Store<List<String>>();
    store.setDataDirectory(MY_BDB_DATA_DIR);
    store.init();
    // save something into the store
    String id = "some_id";
    List<String> values = new ArrayList<String>();
    values.add("value_1");
    values.add("value_2");
    store.setValue(id, values);
    ...
    // retrieve the value from the store
    List<String> retrievedValues = store.getValue(id);
    ...
    // clean up
    store.destroy();
  }
  ...
}

If you have read my earlier blog referenced above, the code here is virtually identical to the code in there. The only difference is the use of generics to make the code reusable regardless of the payload to be persisted, without having to repeat all the boilerplate code that is needed to initialize the BerkeleyDB store.

My next step was to try and make it configurable using Spring, which is where I ran into issues. I wanted the client to be able to configure multiple such stores, each servicing a particular data type (Java objects, custom objects, or collections of either) by specifying the class name of V and the name of the subdirectory where the data should be persisted. Passing in the class name of V was an idea I got from this IBM Developerworks article - "Don't Repeat your DAO".

However, I could not find an easy way to build a Store<Whatever> object using the Class.forName() mechanism, where Whatever could either be a simple Java object, such as String or Integer, or a custom Java object, or a Collection of Java objects or custom objects. Gafter's Gadget looked kind of promising, but wasn't exactly what I was looking for.

From what I have read from other posts on this subject, what I am trying to do is probably impossible in Java at the moment. Basically, using Class.forName() style calls to reflectively build a class instance whose class name is known is not that simple with generic objects. So generics gives you flexibility at compile time, while Class.forName() gives you the same flexibility at run time. Apparently, you can't have your cake and eat it too.

Of course, I could just implement the factory in code, with a Map of store names and corresponding Store implementations, which I could set up at application startup. However, I would rather not do that if I can help it. If anyone knows of a good way to do this, or know of resources you think might help, would appreciate you pointing me at them.

8 comments:

dr3s3/30/2008 9:03 PM
You may want to check out this blog post: http://www.artima.com/weblogs/viewpost.jsp?thread=208860

I haven't tried it but I was also looking for a solution to this problem. Funny enough for the same reason too, DAOs for BDB with DPL.

I'm making the concrete DAOs just implement a getEntityClass() method instead, since there is only one per DAO. I will most likely need a DAO per Entity anyway, since the SecondaryIndex(es) will make up the bulk of my querying.

Here is some of my base class:

package com.mindhaus.schoolhaus.db;

import javax.annotation.PostConstruct;
import javax.persistence.PersistenceException;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Repository;

import com.sleepycat.je.DatabaseException;
import com.sleepycat.persist.EntityStore;
import com.sleepycat.persist.PrimaryIndex;

@Repository
public abstract class BerkeleyDbDataAccessor<T> implements DataAccessor<T> {

@Autowired
protected transient DatabaseManager databaseManager;

protected transient PrimaryIndex<Long, T> entitiesById;

public BerkeleyDbDataAccessor() {
}

@PostConstruct
public void init() throws DatabaseException {

entitiesById = getStore().getPrimaryIndex(
Long.class, getModelClass());

}

public void setDatabaseManager(DatabaseManager databaseManager) {
this.databaseManager = databaseManager;
}

protected EntityStore getStore() {
return databaseManager.getEntityStore();
}

public void create(T entity) {
try {
entitiesById.putNoOverwrite(entity);
} catch (DatabaseException e) {
throw new PersistenceException(e);
}
}

public T update(T entity) {
try {
T old = entitiesById.put(entity);
return old;
} catch (DatabaseException e) {
throw new PersistenceException(e);
}
}

protected abstract Class<T> getEntityClass();

}

And a concrete child:

package com.mindhaus.schoolhaus.db;

import javax.annotation.PostConstruct;

import org.springframework.stereotype.Repository;

import com.mindhaus.schoolhaus.domain.model.Name;
import com.mindhaus.schoolhaus.domain.model.Student;
import com.sleepycat.je.DatabaseException;
import com.sleepycat.persist.SecondaryIndex;

@Repository
public class StudentDataAccessor extends BerkeleyDbDataAccessor<Student> {

SecondaryIndex<Name, Long, Student> studentsByName;
SecondaryIndex<String, Long, Student> studentsByEmail;

public StudentDataAccessor() {

}

@PostConstruct
public void init() throws DatabaseException {
super.init();
studentsByName = getStore().getSecondaryIndex(entitiesById, Name.class, \"name\");
studentsByEmail = getStore().getSecondaryIndex(entitiesById, String.class, \"email\");

}

@Override
protected Class<Student> getEntityClass() {
return Student.class;
}

}
Sujit Pal4/05/2008 2:50 PM
Thanks for the link and your code, dr3s. The link seems to be a combination of Gafter's gadget and manually climbing the class hierarchy tree. Interesting approach, will check it out and see if I can apply it too...
Tej2/26/2010 6:54 AM
Hi Sujit,

Were you able to try the above approach ?

Please provide me with sample code if this is possible.

Also did you try using hyperJaxb for this. I explored it but seems like the only difficulty is how to create BDB specific annotations like @entity, @PrimaryKey through it.

Any suggestions ?

Thanks,
Tej
Sujit Pal3/02/2010 10:33 AM
Hi Tej, no this code never made it past the proof-of-concept stage. We decided to write this functionality using HSQLDB instead because it provides an SQL interface which is more intuitive for maintainers.
Anonymous6/14/2011 12:55 PM
Thanks for this article. It helped me much on building a project :)
Sujit Pal6/18/2011 9:12 PM
Thanks Anonymous, glad it helped you.
Anonymous6/11/2013 12:02 AM
Thanks Sujit.
I needed to implement BerkleyDB 'immediately' in my project. Using the example that you have provided, I was able to do so :)

~Anurag
Sujit Pal6/11/2013 5:44 PM
Cool, glad it helped :-).

Comments are moderated to prevent spam.

Sunday, February 17, 2008

A Generic BerkeleyDB store using DPL

8 comments: