Lexer: Attributes

Note: Attributes with values of character string can contain multiple characters. Each character in the string serves as a distinct character for that type of attribute. For example, if the string *_.- is specified for Printjoins, each individual character in the string (asterisk, underscore, period, hyphen) is treated as a joining character that is included in the index entry for a token in which the character occurs.

Note: Separate topics are provided for MULTI_LEXER lexer attributes (NLS Language, Language Specific Lexer, Alternate Language Value ID) and for information on how to use the MULTI_LEXER lexer.

Note: Any consistency error involving multiple attributes (for example, specifying NO for both index_text and index_themes) is not detected when you set attribute values, but only when you attempt to use the lexer in an index.

Alternate_Spelling: Enables spelling in the specified language.
Values: GERMAN, DANISH, SWEDISH, or blank (default: alternate spelling enabled in all three languages)

Base_Letter: Specifies whether characters that have diacritical marks (umlauts, cedillas, acute accents, etc.) are converted to their base form before being stored in the text index.
Values: YES or NO (default: base-letter conversion disabled)

Note: base_letter is always NO when composite is GERMAN or DUTCH.

Composite: Specifies a language for which composite word indexing is enabled.
Values: GERMAN, DUTCH, or blank (default: composite word indexing disabled)

Note: mixed_case is always YES when composite is GERMAN.

Continuation: Characters that indicate a word continues on the next line and should be indexed as a single token. The most common continuation characters are hyphen (-) and backslash (\).
Values: character string

Endjoins: Characters that, when encountered as the last character in a token, explicitly identify the end of the token. The character, as well as any other startjoins characters that immediately follow it, is included in the Oracle interMedia Text index entry for the token.
Values: character string

The following rules apply to both startjoins and endjoins:

Index_Text: Specifies whether word information is indexed. The index_themes and index_text attributes cannot both be NO.
Values: YES or NO

Index_Themes: Specifies whether theme information is indexed. (Themes are listed in the interMedia Text knowledge base, described in the product documentation.) Enabling index theme information (YES) enables ABOUT operator for queries. The index_themes and index_text attributes cannot both be NO.
Values: YES or NO

Mixed_Case: Specifies whether the lexer stores the tokens in text index entries exactly as they appear in the text (YES), or converts the tokens in text index entries to all uppercase (NO).
Values: YES or NO (default: tokens converted to all uppercase)

Note: Oracle interMedia Text ensures that text queries match the case-sensitivity of the index being queried. As a result, if you enable case-sensitivity for your text index, queries against the index are always case-sensitive.

Note: mixed_case is always YES when composite is GERMAN.

Newline: Characters that indicate the end of a line of text. BASIC_LEXER uses newline characters in conjunction with punctuations and whitespace characters to identify character strings that serve as paragraph delimiters for sentence/paragraph searching.
Values: NEWLINE (\n) (default) or CARRIAGE_RETURN (\r) (for carriage returns)

Numgroup: A single character that, when it appears in a string of digits, indicates that the digits are groupings within a larger single unit. For example, comma (,) might be defined as a numgroup character because it often indicates a grouping of thousands when it appears in a string of digits (example: 25,350). (See also Note for Numjoin.)
Values: single character

Numjoin: Characters that, when they appear in a string of digits, cause Oracle interMedia Text to index the string of digits as a single unit or word. For example, period (.) may be defined as a numjoin character because it often serves as a decimal point when it appears in a string of digits (example: 99.375).
Values: character string

Note: The default values for numjoin and numgroup are determined by the NLS initialization parameters that are specified for the database. In general, a value need not be specified for either numjoin or numgroup when creating a Lexer preference for the BASIC_LEXER object.

Printjoins: Non-alphanumeric characters that, when they appear anywhere in a word (beginning, middle, or end), are processed by Oracle interMedia Text as alphanumeric and included with the token in the text index. This includes printjoins that occur consecutively. For example, if the hyphen (-) and underscore (_) characters are defined as printjoins, terms such as pseudo-intellectual and _file_ are stored in the text index as pseudo-intellectual and _file_.
Values: character string

Note: If a printjoins character is also defined as a punctuations character, the character is only processed as an alphanumeric character if the character immediately following it is a standard alphanumeric character or has been defined as a printjoins or skipjoins character.

Note: printjoins and skipjoins are mutually exclusive. The same characters cannot be specified for both attributes.

Punctuations: Non-alphanumeric characters that, when they appear at the end of a word, indicate the end of a sentence. The defaults are period (.), question mark (?), and exclamation point (!). Characters that are defined as punctuations are removed from a token before text indexing; however, if a punctuations character is also defined as a printjoins character, the character is only removed if it is the last character in the token and it is immediately preceded by the same character.
Values: character string

Skipjoins: Non-alphanumeric characters that, when they appear within a word, identify the word as a single token; however, the characters are not stored with the token in the text index. For example, if the hyphen character (-) is defined as a skipjoins, the word pseudo-intellectual is stored in the text index as pseudointellectual.
Values: character string

Note: printjoins and skipjoins are mutually exclusive. The same characters cannot be specified for both attributes.

Startjoins: Characters that, when encountered as the first character in a token, explicitly identify the start of the token. The character, as well as any other startjoins characters that immediately follow it, is included in the Oracle interMedia Text index entry for the token. In addition, the first startjoins character in a string of startjoins characters implicitly end the previous token.
Values: character string

The following rules apply to both startjoins and endjoins:

Theme_Language: Language used for indexing and generating themes. The French knowledge base and lexicon is public in release 8.1.6. This allows indexing and generation of themes for French documents. By default, the basic lexer chooses the knowledge base to use (French or English) based on the NLS_LANG setting for the session. However, if the NLS_LANG setting does not match your documents (for example, your documents are in French and your environment is English), this will cause the wrong lexicon to be used, or (if your environment is neither French nor English) this will cause an error. If you specify, for example, specify FRENCH for Theme_Language, the lexer will produce French themes regardless of the language environment.
Values: AUTO (default), ENGLISH, or FRENCH

Whitespace: Characters that are treated as blank spaces between tokens. BASIC_LEXER uses whitespace characters in conjunction with punctuations and newline characters to identify character strings that serve as sentence delimiters for sentence/paragraph searching. The predefined, default values for whitespace are 'space' and 'tab'; these values cannot be changed. Specifying characters as whitespace characters adds to these defaults.
Values: character string

Buttons (Apply, Revert, Show SQL/Hide SQL)

See also: Main topics (outline)