Indexing

Indexing in general

In order to provide fast and relevant search, Algolia restructures your data in a special way, via Indexing. The extension does this for you - it runs through all of your data - products, categories, and pages - and indexes your data. These indexes are then uploaded to the Algolia servers, either automatically via the extension’s queue, or manually via the Magento console or the command line. Once pushed to the Algolia server, the indexes go through an indexing process that transforms them into searchable data.

For more information on the Indexing process, check out Algolia’s docs.

If you are having any issues with your data, indexes, or queue, please check our troubleshooting guide

The extension can keep your data (products, categories, pages, etc.) synchronized with Algolia in real time. This provides the best, most up-to-date search experience for your customers.

The extension provides two types of indexing mechanisms:

  • Section re-index
    An entire section of your catalog (products, categories, etc.) is pushed to Algolia and reindexed.

  • Single item re-index
    A single item is pushed to Algolia and reindexed whenever it changes (addition or deletion, update of products or categories, etc.).

By default, these operations run synchronously and the administrator has to wait for them to finish before continuing to work. This is not very convenient and can cause unexpected issues. The indexing queue is designed to alleviate these issues by performing indexing in the background. For more on this, read about the extension’s indexing queue

Products indexing

Exact, up-to-date product data is essential to a successful ecommerce implementation. To accommodate many different product data scenarios, the extension provides a high degree of indexing and search configuration.

Indexable products

Not all products from your Magento store are indexed in Algolia. The extension only indexes products that should appear in an autocomplete menu or instant search page. This ensures the wisest usage of your Algolia records and operations.

The extension indexes only products which are:

  • Visible (catalog, search or both)
  • Enabled
  • Not deleted
  • In-stock (if you only want to show “in-stock” products)

If you’re ever missing a product in your Algolia index, make sure to check that it meets this requirements for indexing.

Go here for more about missing data.

Searchable attributes

You can specify which attributes you want the search to look in. Navigate to System > Configuration > Algolia Search > Products tab to see the configuration.

Here you can find a table of the attributes that you want to send to Algolia. For each attribute you can specify if the attribute is Searchable, Retrievable or Ordered. By default all attributes are set to be searched as Unordered. In general this value is better for relevance and we don’t recommend changing it without a specific reason. Go here for information about Searchable Attributes.

For each attribute you can also specify if you want to index an empty value. Usually the right value is “No”. This comes in handy for attributes to be used for faceting, because you can avoid having a useless value of “No” in the faceting list.

Attributes
Configuration of attributes to index

There are 11 attributes that the extension indexes regardless of what is specified in the table above. These attributes are not “searchable”, but can be used for filtering, sorting, customizing ranking, and building your instant search results. These attributes are:

name Product’s name
url URL address of product in store
visibility_search If the product should be visible in search
visibility_catalog If the product should be visible in catalog
categories Tree of categories where the product belongs
categories_without_path Categories where the product belongs without it’s tree path
thumbnail_url URL of thumbnail image of the product
image_url URL of image of the product
in_stock If the product is on stock
price Price of the product
type_id Type of the product (simple, configurable, bundled, …)

Facets

Facets are the attributes you want to use as filters on an instant search results page. Common facets include price, colours, categories, and brands. What facets are right for you depends on the products you are selling and how your customers filter them.

For each facet you should specify the attribute, the label (what is displayed above the filter) and the type of the facet. Prices and numeric filters will be shown as a slider. Other filters will be shown as a list of checkboxes.

The attributes specified as facets are automatically indexed as retrievable but not searchable. There is no need to specify them in the searchable attributes table. They will automatically be configured in Algolia as attributes for faceting.

There are 3 facets by default - price, categories and color.

Facets
Configuration of facets

Sorting strategies

When you enable the extension’s instant search result page, you may want to offer your users more ways to sort the results. This can be by relevance, popularity, price, age, etc.

Sorting by relevancy is always available and it is the default sort. The other sorting strategies you will specify in Sort Settings. For each strategy you should specify the attribute, sort order (ascending / descending) and the label to be displayed on the instant search page.

The attributes you specify for sorting are automatically indexed as retrievable but not searchable. There is no need to specify them in the searchable attributes table.

There are 3 sorting strategies by default, in addition to relevancy - from lowest price to highest, from highest price to lowest, and from newest to oldest.

Sorting strategies
Configuration of sorting strategies
Each sorting strategy will multiply the number of records that will be indexed in Algolia. This will increase the usage against your Algolia plan. For more information see this FAQ entry.

Index settings

The index settings for products managed by the extension are:

  • searchableAttributes
  • customRanking
  • unretrievableAttributes
  • attributesForFaceting
  • maxValuesPerFacet
  • removeWordsIfNoResults

Additional index settings can be managed in the Algolia dashboard or via the extension’s custom events, using the algolia_products_index_before_set_settings custom event. You can hook into this event and modify the settings programmatically directly from Magento.

Categories indexing

In order to reduce the number of Algolia operations and records, the extension indexes only active categories.

You can choose to index other categories if you’d like them to appear in search results.

Show categories
Show categories that are not included in the navigation menu configuration

Based on this setting, the extension will either index all categories or only the categories that are configured to be included in the navigation menu.

Searchable attributes

You can specify which attributes you want to be searchable in your Algolia indices. To configure searchable attributes navigate to the System > Configuration > Algolia Search > Categories tab.

Here you can find a table of the attributes that you want to send to Algolia. For each attribute you can specify if the attribute is Searchable, Retrievable or Ordered. By default all attributes are set to be searched as Unordered. In general this value is better for relevance and we don’t recommend to change it without a specific reason. Go here for information about Searchable Attributes.

Categories attributes
Configuration of categories' searchable attributes

There are 8 attributes that the extension indexes regardless of what is specified in the table above. These attributes are not “searchable”, but can be used for filtering, sorting, customizing ranking, and building your instant search results. These attributes are:

name Category’s name
url URL address of the category in store
path Tree path to the category
level Level of the category in category tree
include_in_menu If the category should be included in navigation menu
_tags [filled automatically by the extension]
popularity Popularity of the category
product_count Number of products in the category

Index settings

The index settings for categories managed by the extension are:

  • searchableAttributes
  • customRanking
  • unretrievableAttributes

Additional index settings can be managed in the Algolia dashboard or via extension’s algolia_categories_index_before_set_settings custom event. You can hook into this event and modify the settings programmatically directly from Magento.

Pages indexing

The extension supports indexing CMS pages, which will allow your customers to search for them in an autocomplete menu. By default all active pages are indexed, but you are able to exclude any that you don’t want to appear in search results (e.g. error pages).

Excluded pages
Configuration of excluded pages

If you want to disable the indexing of pages, you can remove them from the Additional Sections configuration:

Additional sections
Configuration of additional sections

Indexed attributes

The admin configuration does not support modifying indexable page attributes, but they can be changed programmatically by hooking into the algolia_after_create_page_object event. See events.

Default indexed attributes:

name Name of the page
url URL address of the page
slug Slug of the page
content Textual content of the page

Records in Algolia must be smaller than 10 kilobytes. Therefore, if the content of the page is longer than 10,000 characters the content will not be indexed. In this case, the search will be performed only in the page’s name which could have an impact on relevance.

To learn more about record size limits please see the official Algolia documentation.

Index settings

The extension always sends the same settings to Algolia for the pages index:

  • searchableAttributes
    • unordered(slug)
    • unordered(name)
    • unordered(content)
  • attributesToSnippet
    • content:7

Additional index settings can be managed in the Algolia dashboard or via extension’s algolia_pages_index_before_set_settings custom event. See events. You can hook into this event and modify the settings programmatically directly from Magento.

Suggestions indexing

Each query processed by the Magento backend is stored in the Magento database. The query, number of results, and number of searches of that query are all recorded automatically in the catalogsearch_query by Magento without any involvement from the extension.

Only backend searches are stored in the Magento database. Front-end autocomplete and instant search queries are not stored.

When you enable the indexing of suggestions, the extension fetches queries from catalogsearch_query, filters the results according your settings (minimal number of results, minimal popularity, etc.) and pushes filtered queries into your Algolia suggestions index.

Suggestions configuration
Configuration of suggestions

To ensure good data in your Algolia suggestion index, you must have correct data in your catalogsearch_query table. To do that, you must enable backend search with Algolia. Enable Search and Make SEO request in the Algolia extension configuration in the Magento administration.

When this option is enabled, each backend search will be processed by Algolia and the data in catalogsearch_query will be increased over time.

Suggestions are not indexed automatically by the extension. You will need to trigger a reindex manually or add an entry to a cron table for automatic processing. Here’s an example that would run every hour:

1 * * * * php -f /absolute/path/to/magento/shell/indexer.php -- -reindex search_indexer_suggest

Indexed attributes

The extension does not let you modify indexable attributes via the admin configuration, but they can be changed by hooking your method into to algolia_after_create_suggestion_object event. See events.

Default indexed attributes:

query Suggested search query
number_of_results Number of returned results when backend search was performed
popularity Number of searches with the search query
updated_at Date of the last update

Index settings

The extension always sends the same settings to Algolia for the suggestions index:

  • searchableAttributes
    • unordered(query)
  • customRanking
    • desc(popularity)
    • desc(number_of_results)
  • typoTolerance
    • false
  • attributesToRetrieve
    • query
  • removeWordsIfNoResults
    • lastWords

Additional index settings can be managed in the Algolia dashboard or via the extension’s algolia_suggestions_index_before_set_settings custom event. See events. You can hook into this event and modify the settings programmatically directly from Magento.

Additional sections indexing

In the autocomplete menu you can display other sections like color and brands. This feature requires the instant search page to be enabled. The Additional Sections area requires this to work properly.

To be able to choose an attribute in the Additional Section area, you will need to set it as an attribute for faceting in Algolia.

Configuration of additional sections
Configuration of additional sections

Indexed attributes

The extension does not let you modify indexable attributes via the admin configuration, but they can be changed by hooking your method into the algolia_additional_section_items_before_index event.

Default indexed attributes:

value Value of the attribute (eg. Red, Adidas, XL, …)

Index settings

The extension always sends the same settings to Algolia for the additional sections index:

  • searchableAttributes
    • unordered(value)

The extension does not let you modify indexable attributes via the admin configuration, but they can be changed by hooking your method into the algolia_additional_sections_index_before_set_settings event. See events.