Recoll user manual

Recoll indexing can be performed along two different modes:

The choice between the two methods is mostly a matter of preference, and they can be combined by setting up multiple indexes (ie: use periodic indexing on a big documentation directory, and real time indexing on a small home directory). Monitoring a big file system tree can consume significant system resources.

The choice of method and the parameters used can be configured from the recoll GUI: Preferences → Indexing schedule

The parameters describing what is to be indexed and local preferences are defined in text files contained in a configuration directory.

All parameters have defaults, defined in system-wide files.

Without further configuration, Recoll will index all appropriate files from your home directory, with a reasonable set of defaults.

A default personal configuration directory ($HOME/.recoll/) is created when a Recoll program is first executed. It is possible to create other configuration directories, and use them by setting the RECOLL_CONFDIR environment variable, or giving the -c option to any of the Recoll commands.

In some cases, it may be interesting to index different areas of the file system to separate databases. You can do this by using multiple configuration directories, each indexing a file system area to a specific database. Typically, this would be done to separate personal and shared indexes, or to take advantage of the organization of your data to improve search precision.

The generated indexes can be queried concurrently in a transparent manner.

For index generation, multiple configurations are totally independant from each other. When multiple indexes need to be used for a single search, some parameters should be consistent among the configurations.

Recoll knows about quite a few different document types. The parameters for document types recognition and processing are set in configuration files.

Most file types, like HTML or word processing files, only hold one document. Some file types, like email folders or zip archives, can hold many individually indexed documents, which may themselves be compound ones. Such hierarchies can go quite deep, and Recoll can process, for example, a LibreOffice document stored as an attachment to an email message inside an email folder archived in a zip file...

Recoll indexing processes plain text, HTML, OpenDocument (Open/LibreOffice), email formats, and a few others internally.

Other file types (ie: postscript, pdf, ms-word, rtf ...) need external applications for preprocessing. The list is in the installation section. After every indexing operation, Recoll updates a list of commands that would be needed for indexing existing files types. This list can be displayed by selecting the menu option File → Show Missing Helpers in the recoll GUI. It is stored in the missing text file inside the configuration directory.

By default, Recoll will try to index any file type that it has a way to read. This is sometimes not desirable, and there are ways to either exclude some types, or on the contrary to define a positive list of types to be indexed. In the latter case, any type not in the list will be ignored.

Excluding types can be done by adding name patterns to the skippedNames list, which can be done from the GUI Index configuration menu. It is also possible to exclude a mime type independantly of the file name by associating it with the rclnull filter. This can be done by editing the mimeconf configuration file.

In order to define a positive list, You need to edit the main configuration file (recoll.conf) and set the indexedmimetypes configuration variable. Example:

indexedmimetypes = text/html application/pdf
          

There is no GUI way to do this, because this option runs a bit contrary to Recoll main goal which is to help you find information, independantly of how it may be stored.

In the rare case where the index becomes corrupted (which can signal itself by weird search results or crashes), the index files need to be erased before restarting a clean indexing pass. Just delete the xapiandb directory (see next section), or, alternatively, start the next recollindex with the -z option, which will reset the database before indexing.

Xapian versions usually support several formats for index storage. A given major Xapian version will have a current format, used to create new indexes, and will also support the format from the previous major version.

Xapian will not convert automatically an existing index from the older format to the newer one. If you want to upgrade to the new format, or if a very old index needs to be converted because its format is not supported any more, you will have to explicitly delete the old index, then run a normal indexing process.

Using the -z option to recollindex is not sufficient to change the format, you will have to delete all files inside the index directory (typically ~/.recoll/xapiandb) before starting the indexing.

The Recoll index does not hold copies of the indexed documents. But it does hold enough data to allow for an almost complete reconstruction. If confidential data is indexed, access to the database directory should be restricted.

Recoll (since version 1.4) will create the configuration directory with a mode of 0700 (access by owner only). As the index data directory is by default a sub-directory of the configuration directory, this should result in appropriate protection.

If you use another setup, you should think of the kind of protection you need for your index, set the directory and files access modes appropriately, and also maybe adjust the umask used during index updates.

Multiple Recoll indexes can be created by using several configuration directories which are usually set to index different areas of the file system. A specific index can be selected for updating or searching, using the RECOLL_CONFDIR environment variable or the -c option to recoll and recollindex.

A typical usage scenario for the multiple index feature would be for a system administrator to set up a central index for shared data, that you choose to search or not in addition to your personal data. Of course, there are other possibilities. There are many cases where you know the subset of files that should be searched, and where narrowing the search can improve the results. You can achieve approximately the same effect with the directory filter in advanced search, but multiple indexes will have much better performance and may be worth the trouble.

A recollindex program instance can only update one specific index.

The main index (defined by RECOLL_CONFDIR or -c) is always active. If this is undesirable, you can set up your base configuration to index an empty directory.

The different search interfaces (GUI, command line, ...) have different methods to define the set of indexes to be used, see the appropriate section.

If a set of multiple indexes are to be used together for searches, some configuration parameters must be consistent among the set. These are parameters which need to be the same when indexing and searching. As the parameters come from the main configuration when searching, they need to be compatible with what was set when creating the other indexes (which came from their respective configuration directories).

Most importantly, all indexes to be queried concurrently must have the same option concerning character case and diacritics stripping, but there are other constraints. Most of the relevant parameters are described in the linked section.

As of Recoll version 1.18 you have a choice of building an index with terms stripped of character case and diacritics, or one with raw terms. For a source term of Résumé, the former will store resume, the latter Résumé.

Each type of index allows performing searches insensitive to case and diacritics: with a raw index, the user entry will be expanded to match all case and diacritics variations present in the index. With a stripped index, the search term will be stripped before searching.

A raw index allows for another possibility which a stripped index cannot offer: using case and diacritics to discriminate between terms, returning different results when searching for US and us or resume and résumé. Read the section about search case and diacritics sensitivity for more details.

The type of index to be created is controlled by the indexStripChars configuration variable which can only be changed by editing the configuration file. Any change implies an index reset (not automated by Recoll), and all indexes in a search must be set in the same way (again, not checked by Recoll).

If the indexStripChars is not set, Recoll 1.18 creates a stripped index by default, for compatibility with previous versions.

As a cost for added capability, a raw index will be slightly bigger than a stripped one (around 10%). Also, searches will be more complex, so probably slightly slower, and the feature is still young, so that a certain amount of weirdness cannot be excluded.

Most parameters for a given index configuration can be set from a recoll GUI running on this configuration (either as default, or by setting RECOLL_CONFDIR or the -c option.)

The interface is started from the Preferences → Index Configuration menu entry. It is divided in four tabs, Global parameters, Local parameters, Web history (which is explained in the next section) and Search parameters.

The Global parameters tab allows setting global variables, like the lists of top directories, skipped paths, or stemming languages.

The Local parameters tab allows setting variables that can be redefined for subdirectories. This second tab has an initially empty list of customisation directories, to which you can add. The variables are then set for the currently selected directory (or at the top level if the empty line is selected).

The Search parameters section defines parameters which are used at query time, but are global to an index and affect all search tools, not only the GUI.

The meaning for most entries in the interface is self-evident and documented by a ToolTip popup on the text label. For more detail, you will need to refer to the configuration section of this guide.

The configuration tool normally respects the comments and most of the formatting inside the configuration file, so that it is quite possible to use it on hand-edited files, which you might nevertheless want to backup first...

Indexing is always performed by the recollindex program, which can be started either from the command line or from the File menu in the recoll GUI program. When started from the GUI, the indexing will run on the same configuration recoll was started on. When started from the command line, recollindex will use the RECOLL_CONFDIR variable or accept a -c confdir option to specify a non-default configuration directory.

If the recoll program finds no index when it starts, it will automatically start indexing (except if canceled).

The recollindex indexing process can be interrupted by sending an interrupt (Ctrl-C, SIGINT) or terminate (SIGTERM) signal. Some time may elapse before the process exits, because it needs to properly flush and close the index. This can also be done from the recoll GUI File → Stop Indexing menu entry.

After such an interruption, the index will be somewhat inconsistent because some operations which are normally performed at the end of the indexing pass will have been skipped (for example, the stemming and spelling databases will be inexistant or out of date). You just need to restart indexing at a later time to restore consistency. The indexing will restart at the interruption point (the full file tree will be traversed, but files that were indexed up to the interruption and for which the index is still up to date will not need to be reindexed).

recollindex has a number of other options which are described in its man page. Only a few will be described here.

Option -z will reset the index when starting. This is almost the same as destroying the index files (the nuance is that the Xapian format version will not be changed).

Option -Z will force the update of all documents without resetting the index first. This will not have the "clean start" aspect of -z, but the advantage is that the index will remain available for querying while it is rebuilt, which can be a significant advantage if it is very big (some installations need days for a full index rebuild).

Of special interest also, maybe, are the -i and -f options. -i allows indexing an explicit list of files (given as command line parameters or read on stdin). -f tells recollindex to ignore file selection parameters from the configuration. Together, these options allow building a custom file selection process for some area of the file system, by adding the top directory to the skippedPaths list and using an appropriate file selection method to build the file list to be fed to recollindex -if. Trivial example:

            find . -name indexable.txt -print | recollindex -if
          

recollindex -i will not descend into subdirectories specified as parameters, but just add them as index entries. It is up to the external file selection method to build the complete file list.

The most common way to set up indexing is to have a cron task execute it every night. For example the following crontab entry would do it every day at 3:30AM (supposing recollindex is in your PATH):

30 3 * * * recollindex > /some/tmp/dir/recolltrace 2>&1

Or, using anacron:

1  15  su mylogin -c "recollindex recollindex > /tmp/rcltraceme 2>&1"

As of version 1.17 the Recoll GUI has dialogs to manage crontab entries for recollindex. You can reach them from the Preferences → Indexing Schedule menu. They only work with the good old cron, and do not give access to all features of cron scheduling.

The usual command to edit your crontab is crontab -e (which will usually start the vi editor to edit the file). You may have more sophisticated tools available on your system.

Please be aware that there may be differences between your usual interactive command line environment and the one seen by crontab commands. Especially the PATH variable may be of concern. Please check the crontab manual pages about possible issues.

When using the real time monitor, it may happen that some files need to be indexed, but change so often that they impose an excessive load for the system.

Recoll provides a configuration option to specify the minimum time before which a file, specified by a wildcard pattern, cannot be reindexed. See the mondelaypatterns parameter in the configuration section.

The initial default search mode is Query language. Without special directives, this will look for documents containing all of the search terms (the ones with more terms will get better scores), just like the All terms mode which will ignore such directives. Any term will search for documents where at least one of the terms appear.

The Query Language features are described in a separate section.

All search modes allow wildcards inside terms (*, ?, []). You may want to have a look at the section about wildcards for more information about this.

File name will specifically look for file names. The point of having a separate file name search is that wild card expansion can be performed more efficiently on a small subset of the index (allowing wild cards on the left of terms without excessive penality). Things to know:

You can search for exact phrases (adjacent words in a given order) by enclosing the input inside double quotes. Ex: "virtual reality".

When using a stripped index, character case has no influence on search, except that you can disable stem expansion for any term by capitalizing it. Ie: a search for floor will also normally look for flooring, floored, etc., but a search for Floor will only look for floor, in any character case. Stemming can also be disabled globally in the preferences. When using a raw index, the rules are a bit more complicated.

Recoll remembers the last few searches that you performed. You can use the simple search text entry widget (a combobox) to recall them (click on the thing at the right of the text field). Please note, however, that only the search texts are remembered, not the mode (all/any/file name).

Typing Esc Space while entering a word in the simple search entry will open a window with possible completions for the word. The completions are extracted from the database.

Double-clicking on a word in the result list or a preview window will insert it into the simple search entry field.

You can cut and paste any text into an All terms or Any term search field, punctuation, newlines and all - except for wildcard characters (single ? characters are ok). Recoll will process it and produce a meaningful search. This is what most differentiates this mode from the Query Language mode, where you have to care about the syntax.

You can use the Tools → Advanced search dialog for more complex searches.

After starting a search, a list of results will instantly be displayed in the main list window.

By default, the document list is presented in order of relevance (how well the system estimates that the document matches the query). You can sort the result by ascending or descending date by using the vertical arrows in the toolbar.

Clicking on the Preview link for an entry will open an internal preview window for the document. Further Preview clicks for the same search will open tabs in the existing preview window. You can use Shift+Click to force the creation of another preview window, which may be useful to view the documents side by side. (You can also browse successive results in a single preview window by typing Shift+ArrowUp/Down in the window).

Clicking the Open link will start an external viewer for the document. By default, Recoll lets the desktop choose the appropriate application for most document types (there is a short list of exceptions, see further). If you prefer to completely customize the choice of applications, you can uncheck the Use desktop preferences option in the GUI preferences dialog, and click the Choose editor applications button to adjust the predefined Recoll choices. The tool accepts multiple selections of mime types (e.g. to set up the editor for the dozens of office file types).

Even when Use desktop preferences is checked, there is a small list of exceptions, for mime types where the Recoll choice should override the desktop one. These are applications which are well integrated with Recoll, especially evince for viewing PDF and Postscript files because of its support for opening the document at a specific page and passing a search string as an argument. Of course, you can edit the list (in the GUI preferences) if you would prefer to lose the functionality and use the standard desktop tool.

You may also change the choice of applications by editing the mimeview configuration file if you find this more convenient.

The Preview and Open edit links may not be present for all entries, meaning that Recoll has no configured way to preview a given file type (which was indexed by name only), or no configured external editor for the file type. This can sometimes be adjusted simply by tweaking the mimemap and mimeview configuration files (the latter can be modified with the user preferences dialog).

The format of the result list entries is entirely configurable by using the preference dialog to edit an HTML fragment.

You can click on the Query details link at the top of the results page to see the query actually performed, after stem expansion and other processing.

Double-clicking on any word inside the result list or a preview window will insert it into the simple search text.

The result list is divided into pages (the size of which you can change in the preferences). Use the arrow buttons in the toolbar or the links at the bottom of the page to browse the results.

In Recoll 1.15 and newer, the results can be displayed in spreadsheet-like fashion. You can switch to this presentation by clicking the table-like icon in the toolbar (this is a toggle, click again to restore the list).

Clicking on the column headers will allow sorting by the values in the column. You can click again to invert the order, and use the header right-click menu to reset sorting to the default relevance order (you can also use the sort-by-date arrows to do this).

Both the list and the table display the same underlying results. The sort order set from the table is still active if you switch back to the list mode. You can click twice on a date sort arrow to reset it from there.

The header right-click menu allows adding or deleting columns. The columns can be resized, and their order can be changed (by dragging). All the changes are recorded when you quit recoll

Hovering over a table row will update the detail area at the bottom of the window with the corresponding values. You can click the row to freeze the display. The bottom area is equivalent to a result list paragraph, with links for starting a preview or a native application, and an equivalent right-click menu. Typing Esc (the Escape key) will unfreeze the display.

The default format for the result list entries and the detail area of the result table display an icon for each result document. The icon is either a generic one determined from the MIME type, or a thumbnail of the document appearance. Thumbnails are only displayed if found in the standard freedesktop location, where they would typically have been created by a file manager.

Recoll has no capability to create thumbnails. A relatively simple trick is to use the Open parent document/folder entry in the result list popup menu. This should open a file manager window on the containing directory, which should in turn create the thumbnails (depending on your settings). Restarting the search should then display the thumbnails.

There are also some pointers about thumbnail generation on the Recoll wiki.

The preview window opens when you first click a Preview link inside the result list.

Subsequent preview requests for a given search open new tabs in the existing window (except if you hold the Shift key while clicking which will open a new window for side by side viewing).

Starting another search and requesting a preview will create a new preview window. The old one stays open until you close it.

You can close a preview tab by typing Ctrl-W (Ctrl + W) in the window. Closing the last tab for a window will also close the window.

Of course you can also close a preview window by using the window manager button in the top of the frame.

You can display successive or previous documents from the result list inside a preview tab by typing Shift+Down or Shift+Up (Down and Up are the arrow keys).

A right-click menu in the text area allows switching between displaying the main text or the contents of fields associated to the document (ie: author, abtract, etc.). This is especially useful in cases where the term match did not occur in the main text but in one of the fields. In the case of images, you can switch between three displays: the image itself, the image metadata as extracted by exiftool and the fields, which is the metadata stored in the index.

You can print the current preview window contents by typing Ctrl-P (Ctrl + P) in the window text.

The advanced search dialog helps you build more complex queries without memorizing the search language constructs. It can be opened through the Tools menu or through the main toolbar.

The dialog has two tabs:

Click on the Start Search button in the advanced search dialog, or type Enter in any text field to start the search. The button in the main window always performs a simple search.

Click on the Show query details link at the top of the result page to see the query expansion.

Recoll automatically manages the expansion of search terms to their derivatives (ie: plural/singular, verb inflections). But there are other cases where the exact search term is not known. For example, you may not remember the exact spelling, or only know the beginning of the name.

The term explorer tool (started from the toolbar icon or from the Term explorer entry of the Tools menu) can be used to search the full index terms list. It has three modes of operations:

Note that in cases where Recoll does not know the beginning of the string to search for (ie a wildcard expression like *coll), the expansion can take quite a long time because the full index term list will have to be processed. The expansion is currently limited at 10000 results for wildcards and regular expressions. It is possible to change the limit in the configuration file.

Double-clicking on a term in the result list will insert it into the simple search entry field. You can also cut/paste between the result list and any entry field (the end of lines will be taken care of).

See the section describing the use of multiple indexes for generalities. Only the aspects concerning the recoll GUI are described here.

A recoll program instance is always associated with a specific index, which is the one to be updated when requested from the File menu, but it can use any number of Recoll indexes for searching. The external indexes can be selected through the external indexes tab in the preferences dialog.

Index selection is performed in two phases. A set of all usable indexes must first be defined, and then the subset of indexes to be used for searching. These parameters are retained across program executions (there are kept separately for each Recoll configuration). The set of all indexes is usually quite stable, while the active ones might typically be adjusted quite frequently.

The main index (defined by RECOLL_CONFDIR) is always active. If this is undesirable, you can set up your base configuration to index an empty directory.

When adding a new index to the set, you can select either a Recoll configuration directory, or directly a Xapian index directory. In the first case, the Xapian index directory will be obtained from the selected configuration.

As building the set of all indexes can be a little tedious when done through the user interface, you can use the RECOLL_EXTRA_DBS environment variable to provide an initial set. This might typically be set up by a system administrator so that every user does not have to do it. The variable should define a colon-separated list of index directories, ie:

export RECOLL_EXTRA_DBS=/some/place/xapiandb:/some/other/db

Another environment variable, RECOLL_ACTIVE_EXTRA_DBS allows adding to the active list of indexes. This variable was suggested and implemented by a Recoll user. It is mostly useful if you use scripts to mount external volumes with Recoll indexes. By using RECOLL_EXTRA_DBS and RECOLL_ACTIVE_EXTRA_DBS, you can add and activate the index for the mounted volume when starting recoll.

RECOLL_ACTIVE_EXTRA_DBS is available for Recoll versions 1.17.2 and later. A change was made in the same update so that recoll will automatically deactivate unreachable indexes when starting up.

Documents that you actually view (with the internal preview or an external tool) are entered into the document history, which is remembered.

You can display the history list by using the Tools/Doc History menu entry.

You can erase the document history by using the Erase document history entry in the File menu.

The documents in a result list are normally sorted in order of relevance. It is possible to specify a different sort order, either by using the vertical arrows in the GUI toolbox to sort by date, or switching to the result table display and clicking on any header. The sort order chosen inside the result table remains active if you switch back to the result list, until you click one of the vertical arrows, until both are unchecked (you are back to sort by relevance).

Sort parameters are remembered between program invocations, but result sorting is normally always inactive when the program starts. It is possible to keep the sorting activation state between program invocations by checking the Remember sort activation state option in the preferences.

It is also possible to hide duplicate entries inside the result list (documents with the exact same contents as the displayed one). The test of identity is based on an MD5 hash of the document container, not only of the text contents (so that ie, a text document with an image added will not be a duplicate of the text only). Duplicates hiding is controlled by an entry in the GUI configuration dialog, and is off by default.

As of release 1.19, when a result document does have undisplayed duplicates, a Dups link will be shown with the result list entry. Clicking the link will display the paths (URLs + ipaths) for the duplicate entries.

You can customize some aspects of the search interface by using the GUI configuration entry in the Preferences menu.

There are several tabs in the dialog, dealing with the interface itself, the parameters used for searching and returning results, and what indexes are searched.

User interface parameters: 

Result list parameters: 

Search parameters: 

External indexes: This panel will let you browse for additional indexes that you may want to search. External indexes are designated by their database directory (ie: /home/someothergui/.recoll/xapiandb, /usr/local/recollglobal/xapiandb).

Once entered, the indexes will appear in the External indexes list, and you can chose which ones you want to use at any moment by checking or unchecking their entries.

Your main database (the one the current configuration indexes to), is always implicitly active. If this is not desirable, you can set up your configuration so that it indexes, for example, an empty directory. An alternative indexer may also need to implement a way of purging the index from stale data,

<img src="%I" align="left">%R %S %L &nbsp;&nbsp;<b>%T</b><br>
%M&nbsp;%D&nbsp;&nbsp;&nbsp;<i>%U</i>&nbsp;%i<br>
%A %K

<u><b><a href="P%N">%T</a></b></u><br>
%A<font color=#008000>%U - %S</font> - %L

<img src="%I" align="left">%L <font color="#900000">%R</font>
&nbsp;&nbsp;<b>%T&</b><br>%S&nbsp;
<font color="#808080"><i>%U</i></font>
<table bgcolor="#e0e0e0">
<tr><td><div>%A</div></td></tr>
</table>%K

The Recoll KIO slave allows performing a Recoll search by entering an appropriate URL in a KDE open dialog, or with an HTML-based interface displayed in Konqueror.

The HTML-based interface is similar to the Qt-based interface, but slightly less powerful for now. Its advantage is that you can perform your search while staying fully within the KDE framework: drag and drop from the result list works normally and you have your normal choice of applications for opening files.

The alternative interface uses a directory view of search results. Due to limitations in the current KIO slave interface, it is currently not obviously useful (to me).

The interface is described in more detail inside a help file which you can access by entering recoll:/ inside the konqueror URL line (this works only if the recoll KIO slave has been previously installed).

The instructions for building this module are located in the source tree. See: kde/kio/recoll/00README.txt. Some Linux distributions do package the kio-recoll module, so check before diving into the build process, maybe it's already out there ready for one-click installation.

As a sample application, the Recoll KIO slave could allow preparing a set of HTML documents (for example a manual) so that they become their own search interface inside konqueror.

This can be done by either explicitly inserting <a href="recoll://..."> links around some document areas, or automatically by adding a very small javascript program to the documents, like the following example, which would initiate a search by double-clicking any term:

<script language="JavaScript">
    function recollsearch() {
        var t = document.getSelection();
        window.location.href = 'recoll://search/query?qtp=a&p=0&q=' +
            encodeURIComponent(t);
    }
</script>
 ....
<body ondblclick="recollsearch()">

Some characters are recognized as search modifiers when found immediately after the closing double quote of a phrase, as in "some term"modifierchars. The actual "phrase" can be a single term of course. Supported modifiers:

All words entered in Recoll search fields will be processed for wildcard expansion before the request is finally executed.

The wildcard characters are:

You should be aware of a few things when using wildcards.

Two characters are used to specify that a search hit should occur at the beginning or at the end of the text. ^ at the beginning of a term or phrase constrains the search to happen at the start, $ at the end force it to happen at the end.

As this function is implemented as a phrase search it is possible to specify a maximum distance at which the hit should occur, either through the controls of the advanced search panel, or using the query language, for example, as in:

"^someterm"o10

which would force someterm to be found within 10 terms of the start of the text. This can be combined with a field search as in somefield:"^someterm"o10 or somefield:someterm$.

This feature can also be used with an actual phrase search, but in this case, the distance applies to the whole phrase and anchor, so that, for example, bla bla my unexpected term at the beginning of the text would be a match for "^my term"o5.

Anchored searches can be very useful for searches inside somewhat structured documents like scientific articles, in case explicit metadata has not been supplied (a most frequent case), for example for looking for matches inside the abstract or the list of authors (which occur at the top of the document).

It is surprisingly convenient to be able to show or hide the Recoll GUI with a single keystroke. Recoll comes with a small Python script, based on the libwnck window manager interface library, which will allow you to do just this. The detailed instructions are on this wiki page.

This is probably obsolete now. Anyway:

The Recoll source tree contains the source code to the recoll_applet, a small application derived from the find_applet. This can be used to add a small Recoll launcher to the KDE panel.

The applet is not automatically built with the main Recoll programs, nor is it included with the main source distribution (because the KDE build boilerplate makes it relatively big). You can download its source from the recoll.org download page. Use the omnipotent configure;make;make install incantation to build and install.

You can then add the applet to the panel by right-clicking the panel and choosing the Add applet entry.

The recoll_applet has a small text window where you can type a Recoll query (in query language form), and an icon which can be used to restrict the search to certain types of files. It is quite primitive, and launches a new recoll GUI instance every time (even if it is already running). You may find it useful anyway.

Recoll simple filters are usually shell-scripts, but this is in no way necessary. Extracting the text from the native format is the difficult part. Outputting the format expected by Recoll is trivial. Happily enough, most document formats have translators or text extractors which can be called from the filter. In some cases the output of the translating program is completely appropriate, and no intermediate shell-script is needed.

Filters are called with a single argument which is the source file name. They should output the result to stdout.

When writing a filter, you should decide if it will output plain text or HTML. Plain text is simpler, but you will not be able to add metadata or vary the output character encoding (this will be defined in a configuration file). Additionally, some formatting may be easier to preserve when previewing HTML. Actually the deciding factor is metadata: Recoll has a way to extract metadata from the HTML header and use it for field searches..

The RECOLL_FILTER_FORPREVIEW environment variable (values yes, no) tells the filter if the operation is for indexing or previewing. Some filters use this to output a slightly different format, for example stripping uninteresting repeated keywords (ie: Subject: for email) when indexing. This is not essential.

You should look at one of the simple filters, for example rclps for a starting point.

Don't forget to make your filter executable before testing !

If you can program and want to write an execm filter, it should not be too difficult to make sense of one of the existing modules. For example, look at rclzip which uses Zip file paths as identifiers (ipath), and rclics, which uses an integer index. Also have a look at the comments inside the internfile/mh_execm.h file and possibly at the corresponding module.

execm filters sometimes need to make a choice for the nature of the ipath elements that they use in communication with the indexer. Here are a few guidelines:

In any case, the main goal is that it should be easy for the filter to extract the target document, given the file name and the ipath element.

execm filters will also produce a document with a null ipath element. Depending on the type of document, this may have some associated data (e.g. the body of an email message), or none (typical for an archive file). If it is empty, this document will be useful anyway for some operations, as the parent of the actual data documents.

There are two elements that link a file to the filter which should process it: the association of file to mime type and the association of a mime type with a filter.

The association of files to mime types is mostly based on name suffixes. The types are defined inside the mimemap file. Example:

.doc = application/msword

If no suffix association is found for the file name, Recoll will try to execute the file -i command to determine a mime type.

The association of file types to filters is performed in the mimeconf file. A sample will probably be of better help than a long explanation:

[index]
application/msword = exec antiword -t -i 1 -m UTF-8;\
     mimetype = text/plain ; charset=utf-8

application/ogg = exec rclogg

text/rtf = exec unrtf --nopict --html; charset=iso-8859-1; mimetype=text/html

application/x-chm = execm rclchm

The fragment specifies that:

The output HTML could be very minimal like the following example:

<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html;charset=UTF-8">
  </head>
  <body>
   Some text content
  </body>
</html>
          

You should take care to escape some characters inside the text by transforming them into appropriate entities. At the very minimum, "&" should be transformed into "&", "<" should be transformed into "<". This is not always properly done by translating programs which output HTML, and of course never by those which output plain text.

When encapsulating plain text in an HTML body, the display of a preview may be improved by enclosing the text inside <pre> tags.

The character set needs to be specified in the header. It does not need to be UTF-8 (Recoll will take care of translating it), but it must be accurate for good results.

Recoll will process meta tags inside the header as possible document fields candidates. Documents fields can be processed by the indexer in different ways, for searching or displaying inside query results. This is described in a following section.

By default, the indexer will process the standard header fields if they are present: title, meta/description, and meta/keywords are both indexed and stored for query-time display.

A predefined non-standard meta tag will also be processed by Recoll without further configuration: if a date tag is present and has the right format, it will be used as the document date (for display and sorting), in preference to the file modification date. The date format should be as follows:

<meta name="date" content="YYYY-mm-dd HH:MM:SS">
or
<meta name="date" content="YYYY-mm-ddTHH:MM:SS">
          

Example:

<meta name="date" content="2013-02-24 17:50:00">
          

Filters also have the possibility to "invent" field names. This should also be output as meta tags:

<meta name="somefield" content="Some textual data" />

You can embed HTML markup inside the content of custom fields, for improving the display inside result lists. In this case, add a (wildly non-standard) markup attribute to tell Recoll that the value is HTML and should not be escaped for display.

<meta name="somefield" markup="html" content="Some <i>textual</i> data" />

As written above, the processing of fields is described in a further section.

The indexer will interpret ^L characters in the filter output as indicating page breaks, and will record them. At query time, this allows starting a viewer on the right page for a hit or a snippet. Currently, only the PDF, Postscript and DVI filters generate page breaks.

A few elements in the interface are specific and and need an explanation.

Data for an external indexer, should be stored in a separate index, not the one for the Recoll internal file system indexer, except if the latter is not used at all). The reason is that the main document indexer purge pass would remove all the other indexer's documents, as they were not seen during indexing. The main indexer documents would also probably be a problem for the external indexer purge operation.

If you use a BSD-type port system or a prebuilt package (DEB, RPM, manually or through the system software configuration utility), just follow the usual procedure for your system.

The unpackaged binary versions on the Recoll web site are just compressed tar files of a build tree, where only the useful parts were kept (executables and sample configuration).

The executable binary files are built with a static link to libxapian and libiconv, to make installation easier (no dependencies).

After extracting the tar file, you can proceed with installation as if you had built the package from source (that is, just type make install). The binary trees are built for installation to /usr/local.

If you can install any or all of the following through the package manager for your system, all the better. Especially Qt is a very big piece of software, but you will most probably be able to find a binary package.

You may have to compile Xapian but this is easy.

The shopping list:

Check the Recoll download page for up to date version information.

Recoll has been built on Linux, FreeBSD, Mac OS X, and Solaris, most versions after 2005 should be ok, maybe some older ones too (Solaris 8 is ok). If you build on another system, and need to modify things, I would very much welcome patches.

Configure options: 

Normal procedure:

        cd recoll-xxx
        configure
        make
        (practices usual hardship-repelling invocations)
      

There is little auto-configuration. The configure script will mainly link one of the system-specific files in the mk directory to mk/sysconf. If your system is not known yet, it will tell you as much, and you may want to manually copy and modify one of the existing files (the new file name should be the output of uname -s).

Either type make install or execute recollinstall prefix, in the root of the source tree. This will copy the commands to prefix/bin and the sample configuration files, scripts and other shared data to prefix/share/recoll.

If the installation prefix given to recollinstall is different from either the system default or the value which was specified when executing configure (as in configure --prefix /some/path), you will have to set the RECOLL_DATADIR environment variable to indicate where the shared data is to be found (ie for (ba)sh: export RECOLL_DATADIR=/some/path/share/recoll).

You can then proceed to configuration.

recoll.conf is the main configuration file. It defines things like what to index (top directories and things to ignore), and the default character set to use for document types which do not specify it internally.

The default configuration will index your home directory. If this is not appropriate, start recoll to create a blank configuration, click Cancel, and edit the configuration file before restarting the command. This will start the initial indexing, which may take some time.

Most of the following parameters can be changed from the Index Configuration menu in the recoll interface. Some can only be set by editing the configuration file.

thrQSizes = 2 2 2
thrTCounts =  4 2 1

thrQSizes = 2 -1 -1
thrTCounts =  6 1 1

This file contains information about dynamic fields handling in Recoll. Some very basic fields have hard-wired behaviour, and, mostly, you should not change the original data inside the fields file. But you can create custom fields fitting your data and handle them just like they were native ones.

The fields file has several sections, which each define an aspect of fields processing. Quite often, you'll have to modify several sections to obtain the desired behaviour.

We will only give a short description here, you should refer to the comments inside the file for more detailed information.

Field names should be lowercase alphabetic ASCII.

Here follows a small example of a personal fields file. This would extract a specific email header and use it as a searchable field, with data displayable inside result lists. (Side note: as the email filter does no decoding on the values, only plain ascii headers can be indexed, and only the first occurrence will be used for headers that occur several times).

[prefixes]
# Index mailmytag contents (with the given prefix)
mailmytag = XMTAG

[stored]
# Store mailmytag inside the document data record (so that it can be
# displayed - as %(mailmytag) - in result lists).
mailmytag = 

[mail]
# Extract the X-My-Tag mail header, and use it internally with the
# mailmytag field name
x-my-tag = mailmytag

mimemap specifies the file name extension to mime type mappings.

For file names without an extension, or with an unknown one, the system's file -i command will be executed to determine the mime type (this can be switched off inside the main configuration file).

The mappings can be specified on a per-subtree basis, which may be useful in some cases. Example: gaim logs have a .txt extension but should be handled specially, which is possible because they are usually all located in one place.

mimemap also has a recoll_noindex variable which is a list of suffixes. Matching files will be skipped (which avoids unnecessary decompressions or file executions). This is partially redundant with skippedNames in the main configuration file, with a few differences: it will not affect directories, it cannot be made dependant on the file-system location (it is a configuration-wide parameter), and the file names will still be indexed (not even the file names are indexed for patterns in skippedNames. recoll_noindex is used mostly for things known to be unindexable by a given Recoll version. Having it there avoids cluttering the more user-oriented and locally customized skippedNames.

mimeconf specifies how the different mime types are handled for indexing, and which icons are displayed in the recoll result lists.

Changing the parameters in the [index] section is probably not a good idea except if you are a Recoll developer.

The [icons] section allows you to change the icons which are displayed by recoll in the result lists (the values are the basenames of the png images inside the iconsdir directory (specified in recoll.conf).

mimeview specifies which programs are started when you click on an Open link in a result list. Ie: HTML is normally displayed using firefox, but you may prefer Konqueror, your openoffice.org program might be named oofice instead of openoffice etc.

Changes to this file can be done by direct editing, or through the recoll GUI preferences dialog.

If Use desktop preferences to choose document editor is checked in the Recoll GUI preferences, all mimeview entries will be ignored except the one labelled application/x-all (which is set to use xdg-open by default).

In this case, the xallexcepts top level variable defines a list of mime type exceptions which will be processed according to the local entries instead of being passed to the desktop. This is so that specific Recoll options such as a page number or a search string can be passed to applications that support them, such as the evince viewer.

As for the other configuration files, the normal usage is to have a mimeview inside your own configuration directory, with just the non-default entries, which will override those from the central configuration file.

All viewer definition entries must be placed under a [view] section.

The keys in the file are normally mime types. You can add an application tag to specialize the choice for an area of the filesystem (using a localfields specification in mimeconf). The syntax for the key is mimetype|tag

The nouncompforviewmts entry, (placed at the top level, outside of the [view] section), holds a list of mime types that should not be uncompressed before starting the viewer (if they are found compressed, ie: mydoc.doc.gz).

The right side of each assignment holds a command to be executed for opening the file. The following substitutions are performed:

In addition to the predefined values above, all strings like %(fieldname) will be replaced by the value of the field named fieldname for the document. This could be used in combination with field customisation to help with opening the document.

ptrans specifies query-time path translations. These can be useful in multiple cases.

The file has a section for any index which needs translations, either the main one or additional query indexes. The sections are named with the Xapian index directory names. No slash character should exist at the end of the paths (all comparisons are textual). An exemple should make things sufficiently clear

          [/home/me/.recoll/xapiandb]
          /this/directory/moved = /to/this/place

          [/path/to/additional/xapiandb]
          /server/volume1/docdir = /net/server/volume1/docdir
          /server/volume2/docdir = /net/server/volume2/docdir