Wednesday February 22, 2006

I'd never dug into where velocity's annoying messages were coming from but I decided enough is enough already. These tiresome messages from velocity were showing up on every page load:

2006-02-22 12:08:02 StandardContext[/webapp] Velocity   [info] ResourceManager : found /path/to/resource.vm with loader org.apache.velocity.tools.view.servlet.WebappLoader

resource.manager.logwhenfound = false

They shoulda named that property resource.manager.cmon.feel.the.noise, seriously.

java velocity logging

Monday February 13, 2006

I did a double take on this:

        HashSet set = new HashSet();
        set.add(new URL("http://postsecret.blogspot.com"));
        set.add(new URL("http://dorion.blogspot.com"));
        for (Iterator it = set.iterator(); it.hasNext();) {
            System.out.println(it.next());
        }

http://postsecret.blogspot.com
http://dorion.blogspot.com

http://dorion.blogspot.com
http://postsecret.blogspot.com

http://postsecret.blogspot.com

Hmmm....

The java.net.URL javadoc says what I'd expect "Creates an integer suitable for hash table indexing." So I tried this:

        URL url1 = new URL("http://postsecret.blogspot.com");
        URL url2 = new URL("http://dorion.blogspot.com");
        System.out.println(url1.hashCode() + " " + url1);
        System.out.println(url2.hashCode() + " " + url2); 

1117198397 http://postsecret.blogspot.com
1117198397 http://dorion.blogspot.com

java programming

Wednesday February 08, 2006

One of the really wonderful and evil things about Perl is the tie interface. You get a persistent hash without writing a boat load of code. With Sleepycat's BerkeleyDB Java Edition you can do something very similar.

Here's a quick re-cap: I've mentioned fiddling with BerkeleyDB-JE before with a crude "hello world" app. You can use the native code version with Perl with obscene simplicity, too. In years past, I enjoyed excellent performance with older versions of BerkeleyDB that used a class called "DB_File" -- today, the thing to use is the "BerkeleyDB" library off of CPAN (note, you need db4.x+ something for this to work). Here's a sample that writes to a BDB:

#!/usr/bin/perl

use BerkeleyDB;
use Time::HiRes qw(gettimeofday);
use strict;

my $filename = '/var/tmp/bdbtest';
my %hash = ();
tie(%hash, 'BerkeleyDB::Hash', { -Filename => $filename, -Flags => DB_CREATE });
$hash{'539'} = "d\t" . join('',@{[gettimeofday]}) . "\tu\thttp://www.sifry.com/alerts";
$hash{'540'} = "d\t" . join('',@{[gettimeofday]}) . "\tu\thttp://epeus.blogspot.com";
$hash{'541'} = "d\t" . join('',@{[gettimeofday]}) . "\tu\thttp://http://joi.ito.com";
untie(%hash);

#!/usr/bin/perl

use BerkeleyDB;
use strict;

my $filename = '/var/tmp/bdbtest';
my %hash = ();
tie(%hash, 'BerkeleyDB::Hash', { -Filename => $filename, -Flags => DB_RDONLY });
for my $bid (keys %hash) {
    my %blog = split(/\t/,$hash{$bid});
    print "$bid:\n";
    while(my($k,$v) = each(%blog)) {
        print "\t$k => $v\n";
    }
    
}
untie(%hash); 

541:
        u => http://http://joi.ito.com
        d => 1139388034903283
539:
        u => http://www.sifry.com/alerts
        d => 1139388034902888
540:
        u => http://epeus.blogspot.com
        d => 1139388034903227 

Java has no tie operator (that's probably a good thing). But Sleepycat has incorporated a Collections framework that's pretty cool and gets you pretty close to tied hash functionality. Note however that it's not entirely compatible with the interfaces in the Java Collections Framework but if you know those APIs, you'll immediately know the Sleepycat APIs.

com.sleepycat.collections.StoredMap implements java.util.Map with the folloing cavaets:

1. It doesn't know how big it is, so don't call the size() method unless you want to see a UnsupportedOperationException
2. You can't just abandon java.util.Iterators that have been working on a StoredMap, you have to use com.sleepycat.collections.StoredIterator's .close(Iterator) method to tidy up.

So what does the code look like? Well, let's say you wanted to store a bunch of these vanilla beans in the database:

public final class ImmutableBlog implements Serializable {

    private static final long serialVersionUID = -7882532723565612191L;
    private long lastmodified;
    private String url;
    private int id;
    
    public ImmutableBlog(final int id, final long lastmodified, final String url) {
        this.id = id;
        this.lastmodified = lastmodified;
        this.url = url;
    }
    public int getId() {
        return id;
    }
    public long getLastmodified() {
        return lastmodified;
    }
    public String getUrl() {
        return url;
    }
    
    public boolean equals(Object o) {
        if (!(o instanceof ImmutableBlog))
            return false;
        if (o == this)
            return true;
        ImmutableBlog other = (ImmutableBlog)o;
        return other.getId() == this.getId() &&
            other.getLastmodified() == this.getLastmodified() &&
            other.getUrl().equals(this.getUrl());
    }
    
    public int hashCode() {
        return (int) (id * 51 + url.hashCode() * 17 + lastmodified * 29);
    }
    
    public String toString() {
        StringBuffer sb = new StringBuffer(this.getClass().getName());
        sb.append("[id=")
        .append(id)
        .append(",lastmodified=")
        .append(lastmodified)
        .append(",url=")
        .append(url)
        .append("]");
        return sb.toString();
    }
}

public class StoredBlogMap  {
    
    private StoredMap blogMap;
    public StoredBlogMap() throws Exception {
        init();
    }
    
    protected void init() throws Exception {
        File dir = new File(System.getProperty("java.io.tmpdir") +
                File.separator + "StoredBlogMap");
        dir.mkdirs();
        EnvironmentConfig envConfig = new EnvironmentConfig();
        envConfig.setAllowCreate(true);
        Environment env = new Environment(dir, envConfig);
        DatabaseConfig dbConfig = new DatabaseConfig();
        dbConfig.setAllowCreate(true);
        Database blogsdb = env.openDatabase(null, "blogsdb", dbConfig);
        Database classdb = env.openDatabase(null, "classes", dbConfig);
        StoredClassCatalog catalog = new StoredClassCatalog(classdb);
        blogMap = new StoredMap(blogsdb,
                new IntegerBinding(), new SerialBinding(catalog, 
                        ImmutableBlog.class), true);
    }
    
    public Map getBlogMap() {
        return blogMap;
    }
}

public class StoredBlogMapTest extends TestCase {

    private static Map testMap;
    static {
        testMap = new HashMap();
        testMap.put(new Integer(539), 
                new ImmutableBlog(539, System.currentTimeMillis(), 
                        "http://www.sifry.com/alerts"));
        testMap.put(new Integer(540), 
                new ImmutableBlog(540, System.currentTimeMillis(), 
                        "http://epeus.blogspot.com"));
        testMap.put(new Integer(541), 
                new ImmutableBlog(541, System.currentTimeMillis(), 
                        "http://www.arachna.com/roller/page/spidaman"));        
    };
    private StoredBlogMap blogMap;
    
    protected void setUp() throws Exception {
        super.setUp();
        blogMap = new StoredBlogMap();
    }
    
    public void testWriteBlogs() throws Exception {
        Map blogs = blogMap.getBlogMap();
        for (Iterator iter = testMap.entrySet().iterator(); iter.hasNext();) {
            Map.Entry ent = (Map.Entry) iter.next();
            blogs.put((Integer)ent.getKey(), (ImmutableBlog)ent.getValue());
        }
        int i = 0;
        for (Iterator iter = blogMap.getBlogMap().keySet().iterator(); iter.hasNext();) {
            iter.next();
            i++;
        }
        assertEquals(testMap.size(), i);
    }
    
    public void testReadBlogs() throws Exception {
        Map blogs = blogMap.getBlogMap();
        Iterator iter = blogs.entrySet().iterator();
        while (iter.hasNext()) {
            Map.Entry ent = (Map.Entry) iter.next();
            ImmutableBlog test = (ImmutableBlog) testMap.get(ent.getKey());
            ImmutableBlog stored = (ImmutableBlog) ent.getValue();
            assertEquals(test, stored);
        }
        StoredIterator.close(iter);
    }

    public static void main(String[] args) {
        junit.textui.TestRunner.run(StoredBlogMapTest.class);
    }
}

// cache size is the number of bytes to allow Sleepycat to nail up
envConfig.setCacheSize(cacheSize);
// ... now setup the Environment

The basic stuff here functions very well, however I haven't run the any production code that uses Sleepycat's Collections yet. My last project with BDB needed to run an asynchronous database entry remover, so I wanted to remove as much "padding" as possible.

java sleepycat databases perl

Wednesday February 01, 2006

Note to self: If you're getting OutOfMemoryError's, bumping up the heap size may actually make the problem worse. Usually, OOM means you've exceeded the JVM's capacity... so you set -Xms and -Xmx to a higher strata of memory allocation. Well, at least I thought that was the conventional wisdom. Having cranked it up to 1850M to open very large data structures, OOM's were still bringing down the house. OK, spread the work around in smaller chunks across multiple JVMs. But it still bombs out. It turns out that you have to be very particular about giving the JVM a lot of heap up front. This set of posts seems to peg it. I'd figured that nailing a big heap allocation was how I'd prevent OOM'ing. Looks like it's time for me to bone up on JVM tuning. I should probably dig into 64-bit Java while I'm at it.

java performance tuning 64bit