Importing your catalog from Akeneo to Magento

Ok, I already know that Akeneo rocks. So much that my catalog is already so cool into it. Now I want this catalog in my Magento too. How can I manage that? Oh, Akeneo provides a magento module for doing all the work: akeneo-connector, formerly known as Pimgento:

https://marketplace.akeneo.com/extension/akeneo-connector-magento-2-community-edition

There are many other alternatives, another modules for make a catalog import. And we always can use some generic module, as Akeneo can export the catalog as CSV files. But with Akeneo-Connector we will have the power and confort of Akeneo webservices, in a module maintained by Akeneo itself.

⚠ Technical language Alert

This article is intended for developers who are facing Akeneo_Connector / Pimgento and want to understand how it works.

What are we working on?

In this example we have:

A Akeneo CE 3.2.7 with three channels (website1, website2 and brochures) and two languages (USA English and Spain Spanish).
A Magento CE 2.3.1 with a website (website1 with English language).
Akeneo-Connector v100.3.3
Php 7.2
MySQL 5.7.25

Previous steps

I already have in Akeneo the channels selected for my magento. As I have two websites, the idea is to assign each website to one separate channel.

I’ve configured my Magento with the Akeneo credentials, and I’ve added the needed configuration.

Now I can launch the import. I can do it throw the admin panel or from the server terminal.

But although the providen admin interface is very intuitive, it has a problem. In long imports the process does not end, since each step is validated via AJAX, and if any of them takes more than the maximum waiting time configured in the server (typically 30 seconds), it does not continue with all the steps.

How does it work?

Akeneo-Connector brings all the information directly from Akeneo by webapi (REST). Akeneo has one of the fastest and best documented webapis I’ve ever seen. https://api.akeneo.com/

The import is divided into different stages or phases, which coincide with the types of entities that exist in Akeneo. These stages are launched independently, and are not mandatory, but they must be launched in the correct order.

It installs some auxiliary tables; some are for action log and others are used for some phases, but there is a very important table, “akeneo_connector_entities”. It keeps the relationship between all the entities of Akeneo with those of Magento, associating them by ids. If you ever have any problem of duplicated data on an import, check this table because surely the origin is there.

All phases have a similar operation: they connect to Akeneo to obtain their entities, that are stored in a temporary table (fake temporary, since as long as it exists it will be accessible from any MySQL session), checks the relationship with the Mgento entities in the table “akeneo_connector_entities” and updates it if necessary, and saves the Magento entities with this related data.

The different phases are as follows:

Families

Akeneo families become into Magento attribute-set.
Las familias de akeneo se convierten en Magento en attribute-set. Attribute groups will be imported too. A table “akeneo_connector_family_attribute_relations” is saved with the relationship between akeneo attributes and attribute-set to be used during the attributes import phase.

Attributes

The existing attributes in akeneo are searched and related to those of magento, either by the attribute code itself or by the configured mapping, if it exists. Attributes that are not related will be created in Magento under an equivalence of basic types, or the advanced one if it’s configured.

Tips

If there is an attribute problem, it is most likely some incorrectly related data. Check the table “akeneo_connector_entities” for unexistant magento attributes, or repeated identifiers.
If there is some new attribute created as “static” type, check the previous tip, as something is wrong. What I usually do is to delete the attribute and its relationship in “akeneo_connector_entities” and importing attributes again.

Options

The select and multiselect attribute options must be created separately in Magento, so they are imported in a very similar way to other entities.
There is an exception: if you import Metric attributes as select (used in the variants), the options will be filled during the import of products, since for Akeneo the metric attributes have no options.

Product Models

The “product models” of Akeneo are the equivalent of Magento configurable products. For Magento it is simply a type of products with some related simple products. But for Akeneo it is a grouping of products with a definition of common attributes and other specific attributes of simple products.

Since they are conceptually very different on the two platforms, this step follows a significantly different process from the rest of the entities. A table “akeneo_connector_product_model” is created. It will contain the common attributes and the “axis” attributes on which the configurable product will pivot.

Nothing is done but store this data, since it will be in the import of products where the configurable products will be created into Magento.

Configuration Tip
If you have configurable products, it is very important to configure the mapping of configurable product attributes. Keep in mind that in akeneo an attribute can only be in the product model or in the variant, never in both. So you probably have to devote time and several tests to the configuration:
We’ll choose which attributes must come from the product model, which must be inherited from the first child product, which will have a default value, and even establish an SQL statement on the akeneo-connector temporary tables to obtain the value.

Family variants

Akeneo data regarding family variants is collected, that is, the different combinations of attributes that occur in product models, and the table akeneo_connector_product_model is updated.
Tip

If there are no product models in Akeneo, this import will give an SQL error since the necessary columns will not exist. If you are automating the import, be sure to capture this false error.

Products

Here comes the core of the import. It is divided into the following steps (defined in the code as “jobs”):

createTable: Create temporary table to store data from Akeneo.
insertData: Get the Akeneo data and save it in the temporary table.
addRequiredData: Add in the temporary table necessary data that does not come natively from akeneo. For example, it generates the url-key of the products.
createConfigurable: Use data from simple products and preloaded products in the table akeneo_connector_product_model to build the rows of configurable products in the temporary table.
matchEntities. Compare the rows of the temporary table with existing products by sku and assign the corresponding ids.
updateAttributeSetId. Assign the attribute set to the products in the temporary table.
UpdateOption. Change the value of the select and multiselect type attributes to the corresponding optionId, created in the options import.
createEntities. Create and update magento products.
setValues. Create and update attribute values for imported products.
linkConfigurable. Link configurable products with their children products.
setCategories. Link the products to their categories.
setWebsites. Link the products to their websites. Either according to the configured channel or according to the website attribute if it has been configured.
initStock. Initialize the stock tables for imported new products.
setRelated. Establish relationships between products. Here the relationships of grouped products, related products and cross-selling products are created.
setUrlRewrite. Update the magento url-rewrite for imported products, adapting it to the url-key. Attention, if the url-key field is not mapped, the extension will create it based on the sku.
importMedia. Import the images. Use the akeneo webservice to download the images and assign them to Magento products.
dropTable. Delete the temporary table.
cleanCache. Clean Magento cache.

Configuration Tips
Study the entire product configuration section, as it is quite extensive. Some notes:

Take special care to configure the media attributes. On the one hand you have to tell akeneo-connector which columns must be interpreted as an image, and in another configuration, which one will be saved as image attributes (base, small, thumbnail).
Be careful when configuring the “Default tax class” that establishes the taxclass of the products. It will change the one that has all the products that come from akeneo to this one that is configured.
Take an eye on the attributes type “Metric” of Akeneo. It allows these attributes to be pivotable for the Product Model, but by default they are imported into Magento as a “Text” type, which disables them to be pivotable. If they are to be pivotable, add them to the “Metric Attributes” configuration section as used as a variant.
The attributes of the configurable products are the most delicate sections to configure, devote them a lot of attention and affection.

Tips

Of the jobs that are done, the longest process is usually the regeneration of url-rewrite (apart of downloading thousands of images, of course), especially if you have many stores. Keep this in mind if you are going to automate the process, and if you do it manually, don’t despair. If you are doing test imports and do not need it, disable the generation of url-rewrite in the Akeneo-connector configuration.
Do not configure urls regeneration if you have old url-rewrites. It generates errors by duplicate keys. Usually we launch an external post-import process to regenerate the url-rewrite.

Tiempos

It is surely what worries you most if you have never done this before.

With an Akeneo with a catalog of 17,000 products, without images, the most notable times during the import of products have been:

Get the Akeneo data and save it in the temporary table: 2 minutes and 30 seconds
Fill in the required data: 8 seconds.
Save attribute data: 15 seconds.
Assign products to categories: 5 minutes.
Product relations: 1 minute and 40 seconds.
URL-rewrite: with empty url-rewrites for products, 2 minutes. With previous data up to 15 minutes and duplicate errors.
Images: Naturally it depends on the amount of images to be imported. In this example we have not imported images.

The total import of products has been 15 minutes.

The rest of the tasks and phases are carried out in little remarkable times.

Customization tips

Although Akeneo-connector is very complete and configurable, it is inevitable that it does not fit 100% to your project and surely you have to make some modifications. Here are some tips:

As always in magento 2, to modify existing functionality, first try to do so by Observer. If this is not possible, by plugin after or before, and if it is not possible, I would recommend using a preference although plugins are strongly advised. However a plugin around can block other plugins, but the preference does not.
For very specific and separate functionalities, the cleanest way to extend the import is with a new step. If you want to add an additional task, you will have to make a preference of the specific job, and define the complete task order. In this example is being added a task importFiles after importMedia:

[cc lang=”xml”]

createTable
Create temporary table

insertData
Fill temporary table

addRequiredData
Add required data

createConfigurable
Create configurables

matchEntities
Match code with Magento ID

updateAttributeSetId
Match family code with Magento ID

updateOption
Update column values for options

createEntities
Create or update product entities

setValues
Set values to attributes

linkConfigurable
Link configurable with children

setCategories
Set products to categories

setWebsites
Set products to websites

initStock
Init stock

setRelated
Update related, up-sell and cross-sell products

setUrlRewrite
Set Url Rewrite

importMedia
Import media files

importFiles
Import attribute files

dropTable
Drop temporary table

cleanCache
Clean cache

[/cc]

Conclusion

The basis of working with the Akeneo Connector has already been read several times in the article: pay attention to the configuration. Properly configured the import of the catalog from Akeneo is fast and efficient. And you can fill in the shortcomings you have if you dare to personalize it with your extensions.

Another fight will be that Akeneo’s potential is exploited.