lucene index, NFS, and multiple servers..

Hi, I am trying to get the lucene index to write to a NFS mount, however, it seems to hang the app. If I switch back to local filesystem it works. The problem I have, is in a multi server clustered environment, the indexes get off, and if you round robin a user between servers, the search results are inconsistent.

Anyone else run into this? It seems we can't set many options, except the analzyer it uses, but not the filesystem class. Also looking at the code, it seems if you turn off lucene searching, you don't get any search at all?

Another question - how often does it reindex, does it run a quartz job, and can we set how often it polls if we do? Or do I need to add that?

As I know, adding/editing/deleting a post will modify the lucene index immediately. And reindex is done manually from the Admin Control Panel. There is no quartz job for doing this now.

Maybe you can try to study Apache Solr.

I may try this first, as I think I can get this running for clustered environments quicker than a Solr integration:

http://kalanir.blogspot.com/2008/06/creating-search-index-in-database.html

If that route sucks I'll try messing with Solr. I've gotten everything working pretty well with Terracotta, the lucene reindex issue is the last big issue to tackle before it's solid I think. I'm not trilled with the mods to the *Repository files, but eventually I'd rather have it call the DAO layer directly (per item instead of reloading all of the items) if an item is not in the cache. That however presents some issues, like the other day when the Banlist saving was going in an infinite loop!

I found an article from Sakai Project about Lucene in a cluster:
https://confluence.sakaiproject.org/display/SEARCH/IndexClusterOperation

It seems that the performance issue should be concerned.

I did get the JDBC directory working, and it seems to work quite well BUT there is a major catch:
It needs compass 2.2.0, which is only lucene 2.4.1 compatible. It will not work with lucene 3.3.0.

On my local copy I have jforum rolled back to lucene 2.4.1 with the JDBC directory functioning well.
Another gotcha:
I had to disable the RAMWriter directory. Apparently there is a way to make this work, but I don't quite understand how the RamWriter is used with the 'batchCreate' method and how the flushing to the disk is currently working.

According to this post, there is a way to combine the two:

http://kalanir.blogspot.com/2008/06/loading-index-to-ram-and-flushing.html

But she is doing the 'copy' function, and I -think- jforum is using the index merging, but I'm not sure. I currently added a system global flag to turn it on and off.

Some other catches right now:
1) requires mysql or oracle (only tested on mysql)
2) requires a JNDI datasource. Everything else was a PITA. There might be away around this, but lucene jdbc really wants a datasource in the end anyways

Obviously I'm not checking this in, because I doubt anyways is excited to roll back to lucene 2.4.1. If someone can build compass with Lucene 3.3.0, the problem would be solved (It has seemed Compass has become dormant?)