Major Concepts
This topic covers the following subjects:
- Introduction to Oracle interMedia Text
- Configuring Oracle interMedia Text indexes
- Default preferences used by Oracle interMedia Text
- How Oracle interMedia Text performs indexing (main logic steps)
Introduction to Oracle interMedia Text
The text management component (in previous releases called the Oracle ConText® Cartridge) of the Oracle interMedia product provides a text-search system that allows text to be searched
and managed within the Oracle database as quickly and easily as any other data.
Oracle interMedia Text provides sophisticated text search techniques, making text a
standard datatype in the Oracle server. Using Oracle8i and Oracle interMedia Text, new text-based developments or extensions to existing applications
are easy and cost-effective to build using standard SQL tools.
Application developers can take advantage of Oracle interMedia Text searches in any Oracle database application that uses text, from
search-enabling a comments field in an existing application to implementing
large-scale document management systems dealing with multiple document formats and
complex search criteria. Oracle interMedia Text supports basic full-text search in most languages supported by the
Oracle database.
Configuring Indexes
Documents are stored in many different schemes, formats, and languages.
Consequently, each Oracle interMedia Text index has many options that need to be set to configure it for a
particular situation.
When creating an index, Oracle interMedia Text can use several default values, but most situations require users
to configure the indexes by specifying preferences.
The many options for each index are organized into functional groups, called
"classes", which each concentrate on a particular aspect of the configuration.
You can think of them as asking questions about the document database:
- Data Store: How are the documents stored?
- Filter: How can the documents be converted to plain text?
- Lexer: What language is being indexed?
- Word List: How should stem and fuzzy queries be expanded?
- Storage: Where should the index data be stored?
Each class has a number of pre-defined behaviors called objects. Each object is a possible answer to the class question. For example, the
Data Store class contains the following objects:
- DIRECT_DATASTORE: The documents are stored directly in the column value.
- FILE_DATASTORE: The documents are stored in external files and the column
value is the file name.
- URL_DATASTORE: The documents are stored on Web servers, and the column value
is the URL.
- DETAIL_DATASTORE: The documents are stored in multiple rows of another table.
You can customize preferences to suit your needs. For example, assume that
your documents are stored in the directory /doc
(ctx_ddl.set_attribute('MYPREF','PATH','/doc'). The preference MYPREF specifies that documents are stored as
external files in the directory /doc. To use this preference in index creation,
identify it as the default data store for the desired schema.
A complete list of objects and attributes is contained in the product documentation.
Two groups of options are not configured as preferences:
- Section Group: What is the structure of the document text?
- Stoplist: What are the words which should not be indexed?
Default Preferences
When creating an index, Oracle interMedia Text looks for a preference for each class (data store, filter, section
group, and so on). If no preference is specified by the user, Oracle interMedia Text consults system parameters. Each system parameter contains the name
of the preference to be used as default.
There are system parameters for each class:
- DEFAULT_DATASTORE: Specifies the default data store preference.
- DEFAULT_FILTER_TEXT: Specifies the default filter preference for text column
types (CHAR, VARCHAR2, CLOB, and so on). This is also used when the URL data
store is used in the index.
- DEFAULT_FILTER_BINARY: Specifies the default filter preference for binary
column types (such as RAW or BLOB).
- DEFAULT_FILTER_FILE: Specifies the default filter preference for external
files. This includes BFILE columns and indexes where the file data store is used.
- DEFAULT_SECTION_HTML: Specifies the default section group when the URL data
store or the INSO filter is used.
- DEFAULT_SECTION_TEXT: Specifies the section group to use for all other cases.
- DEFAULT_LEXER: Specifies the default lexer preference.
- DEFAULT_STOPLIST: Specifies the default stoplist.
- DEFAULT_WORDLIST: Specifies the default word list preference.
- DEFAULT_STORAGE: Specifies the default storage preference.
At database creation time Oracle interMedia Text sets these system parameters to appropriate default preferences.
Users, connecting as CTXSYS, can reset these system parameters.
How interMedia Text Performs Indexing
The following are the main logic steps that interMedia Text uses when it indexes a document:
- The data store logic loops over the rows of the table and reads the data out of the column.
Usually, this is simply the column data, but some data stores use the column
data as a pointer to the document data. For example, URL_DATASTORE uses the
column data as a URL.
- The filter takes the document data and transforms it to a text representation. This is
needed when storing binary documents such as Word or Acrobat files. The output
of the filter does not have to be plain text -- it can be a text format such as
XML or HTML.
- The sectioner takes the output from the filter, and converts it to plain text. There are
different sectioners for different text formats, including XML and HTML.
Conversion to plain text includes detecting important section tags, removing
"invisible" information, and reformatting the text.
- The lexer takes the plain text from the sectioner and splits it into discrete tokens.
There lexers for whitespace-delimited languages, as well as specialized lexers
for Asian languages, where segmentation is complex.
- The engine takes all the tokens from the lexer, the section offsets from the
sectioner, and a list of low-information words called stop words, and builds an inverted index. The inverted index stores tokens and the
documents in which these tokens occur.
See also: Main topics (outline)