Search with OpenBD CFML

OpenBD ships with the powerful and popular Apache Lucene search engine in its core distribution and is accessible through a series of specialized tags and functions. Indexing, or building a collection, is done through either passing in query objects, indexing individual files, transversing directories of files, or spidering web sites.

OpenBD supports a wide range of different file formats, enabling the indexing the content locked within each one. These include MS Word documents, PDF, JPG (including all EXIF data), MP3 (ID3 tags), plain and HTML documents.

CFML arranges search groups into collections. A collection can be thought of like a database. Each collection is completely separate from one another and cannot be merged together in searches. A collection contains a series of documents. Each document represents a single unit of searchable content. A document can have many different attributes that you can individually search against. OpenBD places no restrictions on the number of attributes a document can have.

Managing Collections

Collections can be easily manipulated using the inbuilt functions. The collection is stored in a directory on the server, defaults to the "cfcollection" directory under the working directory of OpenBD. A collection has the option of being able to store a complete, unindexed, copy of the content it is indexing. This is something that is not recommended as it takes up a lot of unnecessary space. Instead a collection should be seen like a set of pointers or meta-data to the real data.

Function Name Description
CollectionCreate Creates a brand new collection. If it already exists then this function will throw an exception
CollectionList Lists all the collections registered. A query is returned detailing information about each collection
CollectionDelete Deletes the collection. All files will be removed and the collection will no longer be available for use
CollectionListCategory Lists all the categories this collection uses and the counts for each one. A struct with all the counts are returned
CFCOLLECTION The tag version of the above functions

Below is some sample code to create a new collection.

<cfscript>
CollectionCreate( collection="mycollection", storebody=false );

WriteDump( CollectionList() );
</cfscript>

Adding/Updating/Deleting documents in a collection

A document is a single entity in the collection, much like what a row is to a database. A document contains a number of different attributes and by default must include at least an KEY, which is a unique identifier for that document. If you insert a new document whose KEY matches an existing KEY in the collection, then the original document will be replaced with the new one.

Function Name Description
CollectionIndexCustom Inserts/Updates a document into the collection. The key is the unique identifier for each document. Each field in the document can be searched against. If a query is presented then the fields represent columns into the query. If the column does not exist then an exception is thrown. The index can still be searched while an update is happening, however the new documents will not be available in the search until this operation has completed. Note that all fields are treated as strings and will be indexed accordingly.
CollectionIndexDelete Deletes the given key from the collection. If a query is specified then the key parameter is the column where the unique identifier is. The query is then looped over and all the values deleted from the index
CollectionIndexFile Inserts/Updates a file into the collection. The key is the unique file for each document. Each field in the document can be searched against. If a query is presented then the fields represent columns into the query. If the column does not exist then an exception is thrown. The index can still be searched while an update is happening, however the new documents will not be available in the search until this operation has completed. Note that all fields are treated as strings and will be indexed accordingly.
CollectionIndexPath Inserts/Updates a path into the collection. The key is the unique path for each directory that all files inside will be handled. Each field in the document can be searched against. If a query is presented then the fields represent columns into the query. If the column does not exist then an exception is thrown. The index can still be searched while an update is happening, however the new documents will not be available in the search until this operation has completed. Note that all fields are treated as strings and will be indexed accordingly.
CollectionIndexPurge Removes all the documents in this collection. This does not delete the collection, merely empties the collection ready for further updates
CollectionIndexWeb Inserts/Updates a webpage into the collection. The URL is the unique key for each document. Each field in the document can be searched against. The index can still be searched while an update is happening, however the new documents will not be available in the search until this operation has completed. Note that all fields are treated as strings and will be indexed accordingly. The webpage will be automatically crawled for links and only internal links will be followed. A maximum for 100 URLs will be indexed at once.
CFINDEX The tag version of the above functions

Here is a sample piece of CFML showing how easily it is to index content. OpenBD can recognize and extract information from WORD, PDF, MP3, JPG, TIFF, Plain and HTML files.

<cfscript>
// Indexing a photo folder; subdirectories will be indexed
args = {
  collection : "photocollection",
  recurse : true,
  extensions : ".jpg",
  key : "P:\myphotos\"
};
results = CollectionIndexPath( argumentCollection=args );


// Indexing a website; all internal links will also be indexed
args = {
  collection : "webcollection",
  key : "http://openbd.org/"
};
results = CollectionIndexWeb( argumentCollection=args );


// Indexing a query; each field is a pointer to the column name in the query to pick up the data
args = {
  collection : "customcollection",
  query : myQuery,
  key : "COLUMN_1"
  body : "COLUMN_2,COLUMN3",
  custommap :{
    mycustom : "COLUMN4"
  }
};
results = CollectionIndexCustom( argumentCollection=args );
</cfscript>

Updating and Searching

It is safe to continue searching while a collection is being updated. However, new updated results will not become available until the operation has completed. For example, if you are adding a query for example, or indexing a whole directory.

Searching a collection

Searching against a collection is a very simple manner. Each collection will return a query, with each column representing an attribute in the document. If a document does not have that particular field then the column will be null. You can check for this with the isNull() function.

Function Name Description
CollectionSearch Performs searches against the registered collection using the Apache Lucene internal engine
CFSEARCH The tag version of the above function

Searching can be done using either the tag or the function versions.

<cfset args = {
  collection: "mycollection",
  criteria : "openbd",

  maxrows : 5,
  startrow : 10
}>

<cfset searchresults = CollectionSearch( argumentCollection=args )>

The query string can be built up using a variety of complex types. For example, say you had a custom field indexed as "cartype" then you could perform a search using the criteria="cartype:bmw" which would look in that given custom field for the indexes.

OpenBD has a couple of small tricks to keep the performance and memory usage of searching down. For example, if you wish to have contextpassages returned then you must store the body. However, you generally don't need the full body at that point if you are going to be using the context. So you can set the flag, contents=false, to the CollectionSearch function and the main CONTENTS field will not be returned in the query.

Another performance feature, is the ability to limit the search results. If you have a given collection that has a lot of repeated results. For example, imagine you have indexed all the comments for a blog and when you search you want simply the blog entry, then you can set the unqiuecolumn="blogid" flag on the CollectionSearch function to only return one of the same column. Think of this like a SQL DISTINCT for search.

Apache Lucene Query Syntax Reference

OpenBD does not pre-process the syntax format and simply passes the criteria section straight down to the Lucene drivers. To that end, you can build up complex queries using the Lucene syntax detailed here.

The following content has been reproduced from http://lucene.apache.org/java/2_3_2/queryparsersyntax.html.

Terms

A query is broken up into terms and operators. There are two types of terms: Single Terms and Phrases.

A Single Term is a single word such as "test" or "hello".

A Phrase is a group of words surrounded by double quotes such as "hello dolly".

Multiple terms can be combined together with Boolean operators to form a more complex query (see below).

Note: The analyzer used to create the index will be used on the terms and phrases in the query string. So it is important to choose an analyzer that will not interfere with the terms used in the query string.

Fields

Lucene supports fielded data. When performing a search you can either specify a field, or use the default field. The field names and default field is implementation specific.

You can search any field by typing the field name followed by a colon ":" and then the term you are looking for.

As an example, let's assume a Lucene index contains two fields, title and text and text is the default field. If you want to find the document entitled "The Right Way" which contains the text "don't go this way", you can enter:

title:"The Right Way" AND text:go

or

title:"Do it right" AND right

Since text is the default field, the field indicator is not required.

Note: The field is only valid for the term that it directly precedes, so the query

title:Do it right

Will only find "Do" in the title field. It will find "it" and "right" in the default field (in this case the text field).

Term Modifiers

Lucene supports modifying query terms to provide a wide range of searching options.

Wildcard Searches

Lucene supports single and multiple character wildcard searches within single terms (not within phrase queries).

To perform a single character wildcard search use the "?" symbol.

To perform a multiple character wildcard search use the "*" symbol.

The single character wildcard search looks for terms that match that with the single character replaced. For example, to search for "text" or "test" you can use the search:

te?t

Multiple character wildcard searches looks for 0 or more characters. For example, to search for test, tests or tester, you can use the search:

test*

You can also use the wildcard searches in the middle of a term.

te*t

Note: You cannot use a * or ? symbol as the first character of a search.

Fuzzy Searches

Lucene supports fuzzy searches based on the Levenshtein Distance, or Edit Distance algorithm. To do a fuzzy search use the tilde, "~", symbol at the end of a Single word Term. For example to search for a term similar in spelling to "roam" use the fuzzy search:

roam~

This search will find terms like foam and roams.

Starting with Lucene 1.9 an additional (optional) parameter can specify the required similarity. The value is between 0 and 1, with a value closer to 1 only terms with a higher similarity will be matched. For example:

roam~0.8

The default that is used if the parameter is not given is 0.5.

Proximity Searches

Lucene supports finding words are a within a specific distance away. To do a proximity search use the tilde, "~", symbol at the end of a Phrase. For example to search for a "apache" and "jakarta" within 10 words of each other in a document use the search:

"jakarta apache"~10

Range Searches

Range Queries allow one to match documents whose field(s) values are between the lower and upper bound specified by the Range Query. Range Queries can be inclusive or exclusive of the upper and lower bounds. Sorting is done lexicographically.

mod_date:[20020101 TO 20030101]

This will find documents whose mod_date fields have values between 20020101 and 20030101, inclusive. Note that Range Queries are not reserved for date fields. You could also use range queries with non-date fields:

title:{Aida TO Carmen}

This will find all documents whose titles are between Aida and Carmen, but not including Aida and Carmen.

Inclusive range queries are denoted by square brackets. Exclusive range queries are denoted by curly brackets.

Boosting a Term

Lucene provides the relevance level of matching documents based on the terms found. To boost a term use the caret, "^", symbol with a boost factor (a number) at the end of the term you are searching. The higher the boost factor, the more relevant the term will be.

Boosting allows you to control the relevance of a document by boosting its term. For example, if you are searching for

jakarta apache

and you want the term "jakarta" to be more relevant boost it using the ^ symbol along with the boost factor next to the term. You would type:

jakarta^4 apache

This will make documents with the term jakarta appear more relevant. You can also boost Phrase Terms as in the example:

"jakarta apache"^4 "Apache Lucene"

By default, the boost factor is 1. Although the boost factor must be positive, it can be less than 1 (e.g. 0.2)

Boolean Operators

Boolean operators allow terms to be combined through logic operators. Lucene supports AND, "+", OR, NOT and "-" as Boolean operators(Note: Boolean operators must be ALL CAPS).

OR

The OR operator is the default conjunction operator. This means that if there is no Boolean operator between two terms, the OR operator is used. The OR operator links two terms and finds a matching document if either of the terms exist in a document. This is equivalent to a union using sets. The symbol || can be used in place of the word OR.

To search for documents that contain either "jakarta apache" or just "jakarta" use the query:

"jakarta apache" jakarta

or

"jakarta apache" OR jakarta

AND

The AND operator matches documents where both terms exist anywhere in the text of a single document. This is equivalent to an intersection using sets. The symbol && can be used in place of the word AND.

To search for documents that contain "jakarta apache" and "Apache Lucene" use the query:

"jakarta apache" AND "Apache Lucene"

+

The "+" or required operator requires that the term after the "+" symbol exist somewhere in a the field of a single document.

To search for documents that must contain "jakarta" and may contain "lucene" use the query:

+jakarta lucene

NOT

The NOT operator excludes documents that contain the term after NOT. This is equivalent to a difference using sets. The symbol ! can be used in place of the word NOT.

To search for documents that contain "jakarta apache" but not "Apache Lucene" use the query:

"jakarta apache" NOT "Apache Lucene"

Note: The NOT operator cannot be used with just one term. For example, the following search will return no results:

NOT "jakarta apache"

-

The "-" or prohibit operator excludes documents that contain the term after the "-" symbol.

To search for documents that contain "jakarta apache" but not "Apache Lucene" use the query:

"jakarta apache" -"Apache Lucene"

Grouping

Lucene supports using parentheses to group clauses to form sub queries. This can be very useful if you want to control the boolean logic for a query.

To search for either "jakarta" or "apache" and "website" use the query:

(jakarta OR apache) AND website

This eliminates any confusion and makes sure you that website must exist and either term jakarta or apache may exist.

Field Grouping

Lucene supports using parentheses to group multiple clauses to a single field.

To search for a title that contains both the word "return" and the phrase "pink panther" use the query:

title:(+return +"pink panther")

Escaping Special Characters

Lucene supports escaping special characters that are part of the query syntax. The current list special characters are

+ - && || ! ( ) { } [ ] ^ " ~ * ? : \

To escape these character use the \ before the character. For example to search for (1+1):2 use the query:

\(1\+1\)\:2