www.databasejournal.com/features/mysql/a...

Using Fulltext Indexes in MySQL - Part 1
February 3, 2003

One of the more useful MySQL features is the ability to search for text using a FULLTEXT index. Currently this is only available if you use the MyISAM table type (which is the default table type, so if you don't know what table type you're using, it'll most likely be MyISAM). A fulltext index can be created for a TEXT, CHAR or VARCHAR type field, or combination of fields. We're going to create a sample table and use it to explore the various features.

The simple form of usage (the MATCH() function) is available to all MySQL servers from version 3.23.23, while the more complex usage (the IN BOOLEAN MODE modifier) is available from version 4. The first part of this article looks at the former, and the second part at the latter.
A sample table

We're going to use the following table throughout this tutorial.

CREATE TABLE fulltext_sample(copy TEXT,FULLTEXT(copy)) TYPE=MyISAM;

The TYPE=MyISAM clause isn't necessary unless you've set the default table type to be something other than MyISAM (perhaps you use InnoDB tables to make use of MySQL's transactional capabilities). Once you've created the table, populate it with some data, as follows:

INSERT INTO fulltext_sample VALUES
('It appears good from here'),
('The here and the past'),
('Why are we hear'),
('An all-out alert'),
('All you need is love'),
('A good alert');

If you'd already created an existing table, you can add a FULLTEXT index with the ALTER TABLE statement (as well as the CREATE INDEX statement), for example:

ALTER TABLE fulltext_sample ADD FULLTEXT(copy)

Searching for text

The syntax of a FULLTEXT search is simple. You MATCH the field AGAINST the text you are searching for, for example:

mysql> SELECT * FROM fulltext_sample WHERE MATCH(copy) AGAINST('love');
+----------------------+
| copy |
+----------------------+
| All you need is love |
+----------------------+

Searches on a FULLTEXT index are performed case-insensitively (as are searches on TEXT and non-binary VARCHAR fields generally). So the following works as well:

mysql> SELECT * FROM fulltext_sample WHERE MATCH(copy) AGAINST('LOVE');
+----------------------+
| copy |
+----------------------+
| All you need is love |
+----------------------+

FULLTEXT indexes are most often used to search natural language text, such as through newspaper articles, web page contents and so on. For this reason MySQL has added a number of features to assist this kind of searching. MySQL does not index any words less than or equal to 3 characters in length, nor does it index any words that appear in more than 50% of the rows. This means that if your table contains 2 or less rows, a search on a FULLTEXT index will never return anything. In future, MySQL will make this behavior more flexible, but for now it should suit most natural language uses. If most fields in your database contain the word 'music', you probably don't want these records returned, You can use the IN BOOLEAN MODE modifier to get around the 50% threshold, as you'll see in Part 2 of this article.

Results are returned in order of relevance, from highest to lowest.
The main features

A list of the main features of a standard FULLTEXT search follows:

* Excludes partial words
* Excludes words less than 4 characters in length (3 or less)
* Excludes words that appear in more than half the rows (meaning at least 3 rows are required)
* Hyphenated words are treated as two words
* Rows are returned in order of relevance, descending
* Words in the stopword list (common words) are also excluded from the search results. The stopword list is based upon common English words, so if your data is used for a different purpose, you'll probably want to change the list. Unfortunately, doing so at present is not easy. You'll need to edit the file myisam/ft_static.c. recompile MySQL, and rebuild the indexes! To save you hunting through the source, or if you have a binary version of MySQL, here is a list of stopwords. Note that these can and do change with different versions. To be absolutely sure, you'll have to check the specific list for your version.
Stopwords
"a", "a's", "able", "about", "above", "according", "accordingly", "across", "actually", "after", "afterwards", "again", "against", "ain't", "all", "allow", "allows", "almost", "alone", "along", "already", "also", "although", "always", "am", "among", "amongst", "an", "and", "another", "any", "anybody", "anyhow", "anyone", "anything", "anyway", "anyways", "anywhere", "apart", "appear", "appreciate", "appropriate", "are", "aren't", "around", "as", "aside", "ask", "asking", "associated", "at", "available", "away", "awfully", "b", "be", "became", "because", "become", "becomes", "becoming", "been", "before", "beforehand", "behind", "being", "believe", "below", "beside", "besides", "best", "better", "between", "beyond", "both", "brief", "but", "by", "c", "c'mon", "c's", "came", "can", "can't", "cannot", "cant", "cause", "causes", "certain", "certainly", "changes", "clearly", "co", "com", "come", "comes", "concerning", "consequently", "consider", "considering", "contain", "containing", "contains", "corresponding", "could", "couldn't", "course", "currently", "d", "definitely", "described", "despite", "did", "didn't", "different", "do", "does", "doesn't", "doing", "don't", "done", "down", "downwards", "during", "e", "each", "edu", "eg", "eight", "either", "else", "elsewhere", "enough", "entirely", "especially", "et", "etc", "even", "ever", "every", "everybody", "everyone", "everything", "everywhere", "ex", "exactly", "example", "except", "f", "far", "few", "fifth", "first", "five", "followed", "following", "follows", "for", "former", "formerly", "forth", "four", "from", "further", "furthermore", "g", "get", "gets", "getting", "given", "gives", "go", "goes", "going", "gone", "got", "gotten", "greetings", "h", "had", "hadn't", "happens", "hardly", "has", "hasn't", "have", "haven't", "having", "he", "he's", "hello", "help", "hence", "her", "here", "here's", "hereafter", "hereby", "herein", "hereupon", "hers", "herself", "hi", "him", "himself", "his", "hither", "hopefully", "how", "howbeit", "however", "i", "i'd", "i'll", "i'm", "i've", "ie", "if", "ignored", "immediate", "in", "inasmuch", "inc", "indeed", "indicate", "indicated", "indicates", "inner", "insofar", "instead", "into", "inward", "is", "isn't", "it", "it'd", "it'll", "it's", "its", "itself", "j", "just", "k", "keep", "keeps", "kept", "know", "knows", "known", "l", "last", "lately", "later", "latter", "latterly", "least", "less", "lest", "let", "let's", "like", "liked", "likely", "little", "look", "looking", "looks", "ltd", "m", "mainly", "many", "may", "maybe", "me", "mean", "meanwhile", "merely", "might", "more", "moreover", "most", "mostly", "much", "must", "my", "myself", "n", "name", "namely", "nd", "near", "nearly", "necessary", "need", "needs", "neither", "never", "nevertheless", "new", "next", "nine", "no", "nobody", "non", "none", "noone", "nor", "normally", "not", "nothing", "novel", "now", "nowhere", "o", "obviously", "of", "off", "often", "oh", "ok", "okay", "old", "on", "once", "one", "ones", "only", "onto", "or", "other", "others", "otherwise", "ought", "our", "ours", "ourselves", "out", "outside", "over", "overall", "own", "p", "particular", "particularly", "per", "perhaps", "placed", "please", "plus", "possible", "presumably", "probably", "provides", "q", "que", "quite", "qv", "r", "rather", "rd", "re", "really", "reasonably", "regarding", "regardless", "regards", "relatively", "respectively", "right", "s", "said", "same", "saw", "say", "saying", "says", "second", "secondly", "see", "seeing", "seem", "seemed", "seeming", "seems", "seen", "self", "selves", "sensible", "sent", "serious", "seriously", "seven", "several", "shall", "she", "should", "shouldn't", "since", "six", "so", "some", "somebody", "somehow", "someone", "something", "sometime", "sometimes", "somewhat", "somewhere", "soon", "sorry", "specified", "specify", "specifying", "still", "sub", "such", "sup", "sure", "t", "t's", "take", "taken", "tell", "tends", "th", "than", "thank", "thanks", "thanx", "that", "that's", "thats", "the", "their", "theirs", "them", "themselves", "then", "thence", "there", "there's", "thereafter", "thereby", "therefore", "therein", "theres", "thereupon", "these", "they", "they'd", "they'll", "they're", "they've", "think", "third", "this", "thorough", "thoroughly", "those", "though", "three", "through", "throughout", "thru", "thus", "to", "together", "too", "took", "toward", "towards", "tried", "tries", "truly", "try", "trying", "twice", "two", "u", "un", "under", "unfortunately", "unless", "unlikely", "until", "unto", "up", "upon", "us", "use", "used", "useful", "uses", "using", "usually", "v", "value", "various", "very", "via", "viz", "vs", "w", "want", "wants", "was", "wasn't", "way", "we", "we'd", "we'll", "we're", "we've", "welcome", "well", "went", "were", "weren't", "what", "what's", "whatever", "when", "whence", "whenever", "where", "where's", "whereafter", "whereas", "whereby", "wherein", "whereupon", "wherever", "whether", "which", "while", "whither", "who", "who's", "whoever", "whole", "whom", "whose", "why", "will", "willing", "wish", "with", "within", "without", "won't", "wonder", "would", "would", "wouldn't", "x", "y", "yes", "yet", "you", "you'd", "you'll", "you're", "you've", "your", "yours", "yourself", "yourselves", "z", "zero",



Let's have a look at some of the consequences of this. If you were a bit lazy in your typing, and tried 'to look for the word 'love', as follows:

mysql> SELECT * FROM fulltext_sample WHERE MATCH(copy) AGAINST('lov');
Empty set (0.00 sec)

you'd get nothing back, as the FULLTEXT index only contains complete words, not partial words. You'd have to write the full word to get anything back, as you did in the first example.

As mentioned, hyphenated words are also excluded from the FULLTEXT index (they are indexed as separate words), so the following also returns nothing:

mysql> SELECT * FROM fulltext_sample WHERE MATCH(copy) AGAINST('all-out');
Empty set (0.00 sec)

Unfortunately, both words are less than the required 4 letters, so they don't appear on their own either, and cannot be found at all with an ordinary search. Part 2 of this tutorial looks at BOOLEAN MODE searches when you can search for partial or hyphenated words.

You can also search for more than one word at a time, by separating the words with commas. Try and return records containing the words 'here' and 'appears', as follows:

mysql> SELECT * FROM fulltext_sample WHERE MATCH(copy) AGAINST('here');
Empty set (0.01 sec)

Unexpectedly this returns nothing. But, a more careful look at the stopword list shows this word is listed, so they are excluded from the index. Stopwords are a common cause of people complaining that the MySQL FULLTEXT index facility is not working properly. If your query does return a result, then the stopword list in your version of MySQL does not contain the word 'here'.
Relevance

The following example shows how the records are returned in order of precedence:

mysql> SELECT * FROM fulltext_sample WHERE MATCH(copy) AGAINST('good,alert');
+---------------------------+
| copy |
+---------------------------+
| A good alert |
| It appears good from here |
| An all-out alert |
+---------------------------+

The record 'A good alert' appears first, as it contains both words being searched for. You don't have to believe me - just ask MySQL to display the precedence in the results. Simply repeat the MATCH() function in the field list, as follows:

mysql> SELECT copy,MATCH(copy) AGAINST('good,alert') AS relevance
FROM fulltext_sample WHERE MATCH(copy) AGAINST('good,alert');
+---------------------------+------------------+
| copy | relevance |
+---------------------------+------------------+
| A good alert | 1.3551264824316 |
| An all-out alert | 0.68526663197496 |
| It appears good from hear | 0.67003110026735 |
+---------------------------+------------------+

The relevance calculation is fairly complex and is based upon the number of words in the index, the number of unique words in that row, the total number of words in both the index and the result, as well as the weight of the word (for the average English sentence, the word 'cool' will be weighted less, and therefore have a lower relevance, than the word 'dandy', though trends can change!). The figures may differ in your version of MySQL, as MySQL does occasionally tweak the calculation algorithms.

While the standard FULLTEXT search is fairly useful and sufficient for many, MySQL 4 takes it much further. Part 2 of this article looks at BOOLEAN FULLTEXT searches, which offer a lot more functionality.

Using Fulltext Indexes in MySQL - Part 2, Boolean searches
By Ian Gilfillan

Part 1 of this article looked at the fulltext index, and how to search on it using an ordinary MATCH() AGAINST(). Even more powerful, (although only available on the newer MySQL version 4), is the ability to do a boolean search. Part 2 of this article examines the possibilities.

You'll use the same table you used in Part 1. The full list of records is:

mysql> SELECT * FROM fulltext_sample;
+---------------------------+
| copy |
+---------------------------+
| It appears good from here |
| The here and the past |
| Why are we here |
| An all-out alert |
| All you need is love |
| A good alert |
+---------------------------+

To perform a boolean search, the IN BOOLEAN MODE modifier is used. The following query demonstrates the new syntax.

mysql> SELECT * FROM fulltext_sample
WHERE MATCH (copy) AGAINST ('love' IN BOOLEAN MODE);
+----------------------+
| copy |
+----------------------+
| All you need is love |
+----------------------+

An important difference between ordinary MATCH() AGAINST() queries and IN BOOLEAN MODE searches is that the latter does not apply the 50% limit (so the word can appear in more than half the rows).

There are a number of special operators you'll need to know in order to make the most of the boolean search. If no symbol is specified, the appearance of this word causes the relevance of the row to be higher, similar to an ordinary MATCH() AGAINST().
Boolean Search Operators
+ The word is mandatory in all rows returned.
- The word cannot appear in any row returned.
< The word that follows has a lower relevance than other words, although rows containing it will still match
> The word that follows has a higher relevance than other words.
() Used to group words into subexpressions.
~ The word following contributes negatively to the relevance of the row (which is different to the '-' operator, which specifically excludes the word, or the '<' operator, which still causes the word to contribute positively to the relevance of the row.
* The wildcard, indicating zero or more characters. It can only appear at the end of a word.
" Anything enclosed in the double quotes is taken as a whole (so you can match phrases, for example).
Examples
Let's see some of this action. The first example returns all rows containing the word 'here', but not the word 'past'.

mysql> SELECT copy FROM fulltext_sample WHERE MATCH(copy)
AGAINST ('+here -past' IN BOOLEAN MODE);
+---------------------------+
| copy |
+---------------------------+
| It appears good from here |
| Why are we here |
+---------------------------+

Compare the difference between the next two examples:

mysql> SELECT copy FROM fulltext_sample WHERE MATCH(copy)
AGAINST ('here past' IN BOOLEAN MODE);
+---------------------------+
| copy |
+---------------------------+
| It appears good from here |
| The here and the past |
| Why are we here |
+---------------------------+

mysql> SELECT copy FROM fulltext_sample WHERE MATCH(copy)
AGAINST ('here +past'IN BOOLEAN MODE);
+-----------------------+
| copy |
+-----------------------+
| The here and the past |
+-----------------------+

The first works in a similar way to an ordinary MATCH() AGAINST(), albeit without the 50% threshold. The word 'past' is mandatory in the second search (by default a word is optional), so the other two records are not returned.

The next example demonstrates a common mistake.

mysql> SELECT copy FROM fulltext_sample
WHERE MATCH(copy) AGAINST ('+are here' IN BOOLEAN MODE);
+---------------------------+
| copy |
+---------------------------+
| It appears good from here |
| The here and the past |
| Why are we here |
+---------------------------+

The results may appear surprising compared the previous example, but since 'are' contains three or less letters, it is excluded for purposes of the search and is not mandatory.

The next two examples demonstrate a powerful enhancement for searching purposes:

mysql> SELECT copy FROM fulltext_sample
WHERE MATCH(copy) AGAINST ('aler' IN BOOLEAN MODE);
Empty set (0.01 sec)

mysql> SELECT copy FROM fulltext_sample
WHERE MATCH(copy) AGAINST ('aler*' IN BOOLEAN MODE);
+------------------+
| copy |
+------------------+
| An all-out alert |
| A good alert |
+------------------+

By default only whole words are matched, unless the '*' operator is used.

The '<' and '>' symbols are less commonly used, but they allow a great degree of control for relevance. In the following examples, we return the relevance indicator to demonstrate the difference between the queries.

mysql> SELECT copy, MATCH(copy)
AGAINST ('appears good alert' IN BOOLEAN MODE)
AS m FROM fulltext_sample WHERE MATCH(copy)
AGAINST ('appears good alert' IN BOOLEAN MODE);
+---------------------------+------+
| copy | m |
+---------------------------+------+
| It appears good from here | 2 |
| An all-out alert | 1 |
| A good alert | 2 |
+---------------------------+------+

mysql> SELECT copy, MATCH(copy)
AGAINST ('appears <good alert' IN BOOLEAN MODE)
AS m FROM fulltext_sample WHERE MATCH(copy)
AGAINST ('appears <good alert' IN BOOLEAN MODE);
+---------------------------+------------------+
| copy | m |
+---------------------------+------------------+
| It appears good from here | 1.6666667461395 |
| An all-out alert | 1 |
| A good alert | 1.6666667461395 |
+---------------------------+------------------+

The '<' operator decreases the relevance of the word 'good', in this case by approximately 0.33.

mysql> SELECT copy, MATCH(copy)
AGAINST ('appears good >alert' IN BOOLEAN MODE)
AS m FROM fulltext_sample WHERE MATCH(copy)
AGAINST ('appears good >alert' IN BOOLEAN MODE);
+---------------------------+------+
| copy | m |
+---------------------------+------+
| It appears good from here | 2 |
| An all-out alert | 1.5 |
| A good alert | 2.5 |
+---------------------------+------+

The '>' operator increases the relevance of the word 'alert', in this case by 0.5.

The parentheses group words into a subexpression. In the following example, the '+' symbol applies to the group of words, so that at least one of 'appears' or 'past' must appear.

mysql> SELECT copy, MATCH(copy)
AGAINST ('+(appears past)' IN BOOLEAN MODE)
AS m FROM fulltext_sample WHERE MATCH(copy)
AGAINST ('(appears past)' IN BOOLEAN MODE);
+---------------------------+------+
| copy | m |
+---------------------------+------+
| It appears good from here | 1 |
| The here and the past | 1 |
+---------------------------+------+

You can also apply the operators to words in the subexpression, as follows:

mysql> SELECT copy, MATCH(copy)
AGAINST ('+(>appears <past)' IN BOOLEAN MODE)
AS m FROM fulltext_sample WHERE MATCH(copy)
AGAINST ('+(>appears <past)' IN BOOLEAN MODE);
+---------------------------+------------------+
| copy | m |
+---------------------------+------------------+
| It appears good from here | 1.5 |
| The here and the past | 0.66666668653488 |
+---------------------------+------------------+

The '~' operator contributes negatively to the relevance, but does not bar the word from appearing, as the '-' operator does. It also does not do the same as the '<' operator, which is to make a reduced, yet still positive, difference to the relevance. These subtle difference allows for powerfully tuned searches, but can cause confusion. Look at the difference between the following. First, the word 'here' is compulsory, and the word 'past' optional:

mysql> SELECT copy, MATCH(copy)
AGAINST ('+here past' IN BOOLEAN MODE)
AS m FROM fulltext_sample WHERE MATCH(copy)
AGAINST ('+here past' IN BOOLEAN MODE);
+---------------------------+-----------------+
| copy | m |
+---------------------------+-----------------+
| It appears good from here | 1 |
| The here and the past | 1.3333333730698 |
| Why are we here | 1 |
+---------------------------+-----------------+

Next, a reduced relevance for the word 'past'. The same results are returned, but the record with both words ('The here and the past') has less weighting, though still more than the other records.

mysql> SELECT copy, MATCH(copy)
AGAINST ('+here <past' IN BOOLEAN MODE)
AS m FROM fulltext_sample WHERE MATCH(copy)
AGAINST ('+here <past' IN BOOLEAN MODE);
+---------------------------+-----------------+
| copy | m |
+---------------------------+-----------------+
| It appears good from here | 1 |
| The here and the past | 1.2222222089767 |
| Why are we here | 1 |
+---------------------------+-----------------+

Take note of the relevance in the next example:

mysql> SELECT copy, MATCH(copy)
AGAINST ('+here ~past' IN BOOLEAN MODE)
AS m FROM fulltext_sample WHERE MATCH(copy)
AGAINST ('+here ~past' IN BOOLEAN MODE);
+---------------------------+------------------+
| copy | m |
+---------------------------+------------------+
| It appears good from here | 1 |
| The here and the past | 0.83333331346512 |
| Why are we here | 1 |
+---------------------------+------------------+

The important difference is that the relevance is negatively affected, and is now lower than the other two records. This means if you asked for only '~past', you'd get nothing back, as the relevance would be lower than 0 (and therefore not returned)

mysql> SELECT copy, MATCH(copy)
AGAINST ('~past' IN BOOLEAN MODE)
AS m FROM fulltext_sample WHERE MATCH(copy)
AGAINST ('~past' IN BOOLEAN MODE);
Empty set (0.00 sec)

Finally, '-' simply removes the second record from the result set, as we've seen before.

mysql> SELECT copy, MATCH(copy)
AGAINST ('+here -past' IN BOOLEAN MODE)
AS m FROM fulltext_sample WHERE MATCH(copy)
AGAINST ('+here -past' IN BOOLEAN MODE);
+---------------------------+------+
| copy | m |
+---------------------------+------+
| It appears good from here | 1 |
| Why are we here | 1 |
+---------------------------+------+

The final operator, the double quotes ("), is useful to group phrases together. So, if you were interested in returning 'good from here' but none of those words if they did not appear in the phrase, you'd use:

mysql> SELECT copy, MATCH(copy)
AGAINST ('"good from here"' IN BOOLEAN MODE)
AS m FROM fulltext_sample WHERE MATCH(copy)
AGAINST ('"good from here"' IN BOOLEAN MODE);
+---------------------------+------+
| copy | m |
+---------------------------+------+
| It appears good from here | 1 |
+---------------------------+------+

For many applications, boolean searches are one of MySQL 4's most useful features. With careful use, you can save yourself substantial of development time, and add many useful enhancements. Good luck!

Open_tables, Opened_tables

This statistic shows you the number of tables currently open, while the Opened_tables variables shows the number of tables opened since the server was last restarted. These values provide a good indication of whether your table cache is of the right size; a high value for Opened_tables indicates that your cache should be larger.

Slow_queries

This statistic indicates the number of queries that have taken longer than a predefined per-query time limit. A large value here indicates that the server is not able to process queries at the speed it should and is a cause for concern (although you should make this determination after checking the slow query log, which contains the actual query strings, to see if it's the queries themselves that are unusually long or complex).

Select_scan


This statistic shows the number of joins which required a full scan of the first table named in the join. Since a full table scan is time- and resource-intensive, a high value here means that your queries are operating inefficiently and perhaps require further optimization.

Select_full_join

This statistic reveals the total number of joins performed without making use of indexes. An index speeds up table searches and is advisable on all fields that are queried frequently. Therefore, a high value here means that MySQL is not using indexes and is therefore taking longer to build a result set. The problem can be corrected (or at least mitigated) by indexing important fields of the join.

Qcache_hits


This statistic shows the number of accesses to the new MySQL query cache. The query cache stores the results of frequently-used queries so as to speed up response time; a high value here means that MySQL is working efficiently by using the cache instead of rebuilding result sets every time the same query is fired.