It’s fast to add documents. It doesn’t allow you to add terms, but I don’t know what you mean by adding a term without a doc. You can get the (term,doc,count) pairs through an iterator of TermDocs, which contain terms, docs and frequency counts.

What’s nice about its implementation is that it compresses the reverse index reasonably well and allows you to run either in memory or using memory mapped disk (on top of their Directory base class), so it’s easy to move between efficient and scalable. (Though on-disk is surprisingly efficient because much of the indexing is in memory.)

Removal is fast, but optimizing the index after a bunch of add/deletes so it’s most space efficient and efficient for search is much slower — it basically applies a merge over the whole on-disk index.

It’s also fielded, but it sounds like you don’t need this, so you could just use one field for all terms.

It obviously also does full doc search with a pretty flexible query language and it’s pretty fast. It also lets you easily retrieve the term vectors for the documents.

Lucene also allows for easily pluggable normalizers/tokenizers by writing an Analyzer implementation or using the existing ones.

Also, for the more rule-based crowd, the machine translation system Apertium is all based on the command-line (the main program is just a short shell script calling each module of a language pair pipeline); as is the vislcg3 Constraint Grammar parser.

http://www2.research.att.com/~bs/bs_faq.html#really-say-that