Resolved Problem importing data from ICECAT

Hello,

We have been through this before if I remember correctly. First thing to check would be, what happens if you import that product alone? Does it also not receive any data? The one thing that comes to mind is that there are too many requests from your server to ICEcat and that ICEcat stops responding for a moment.

Do that 1 product import and let's see how that goes.

Dear Roland
I did as you asked.
I only put two products, both present in the ICECAT file "files.index.csv". One is ok but the one indicated in the previous post does not download the data. (Received ICEcat data: [])
There was no active cron job and nothing that can work with my and ICECAT's server
I attach the log file

Thanks

Dear

If it can be useful I attach the two lines of the products in question, extracted from the file "files.index.csv"

Hello,

Thank you for the provided information. So I had a look at the debug log you provided and decided to create my own import file.

So I created a super simple import file with just these lines and set the template to use file for configuration:

product_sku,manufacturer_name,product_name

ITGCADCDSB,ITEK,Test

That loads the ICEcat data as expected.
Now I had a further look at the log and we can see these lines:

• Find the ICEcat ID for manufacturer ITEK and part number ITGCADCDSB
  
• Find the ICEcat ID by similar SKU for manufacturer ITEK and part number ITGCADCDSB
  
• Find the ICEcat ID as alternative ID for manufacturer ITEK and part number ITGCADCDSB
  

This means that RO CSVI cannot find the ICEcat URL. This is why you get no data back. One thing we need to check is if your supplier file has a supplier name ITEK and what is the supplier_id? In my case and in the file you provided this ID is 5237. Now I think that your supplier list has a different ID for ITEK, can you check if that is the case?

That is the only reason I can think of why it would not load the data or that the ITEK supplier does not exist.

Let me know what you find.

Dear Roland

I think you didn't see the file I sent you "righe_index.txt"
Inside the file there are two lines copied from the file "file.index.csv"
The two lines correspond to the products used for the test.
In the file there is the product ITGCACOLVB manufacturer ITEK (Line 1) and this has no problems downloading the data from icecat.
Then there is the product ITGCADCDSB manufacturer ITEK (Line 2) this is the one that has problems.
As you will see both have the supplier_id = 5237 = ITEK.

There is a field, on_market (field to the right of the EAN/UTC code field), which I don't know what it's for, I see that there is a 1 in the ok product while it is 0 in the product with the problem. No I think this field is important

Thanks

Hello,

I definitely saw the righe_index.txt file, as I still have it open on my desktop As I mentioned before, the log shows that RO CSVI cannot find a reference to your second product, that is why no data is available. RO CSVI does not even try to contact ICEcat because it has no URL.

For the first product it immediately finds a reference so it can call ICEcat. That is why your log only shows this:

Find the ICEcat ID for manufacturer ITEK and part number ITGCACOLVB

while for the second product it tries 3 times to find a corresponding URL but finds none.

The first product is found using this query:

SELECT product_id FROM `jos_csvi_icecat_index` AS `i` LEFT JOIN `jos_csvi_icecat_suppliers` AS `s` ON `s`.`supplier_id` = `i`.`supplier_id` WHERE `i`.`prod_id` = 'ITGCACOLVB' AND `s`.`supplier_name` = 'ITEK'

RO CSVI is trying to find the second product using this query

SELECT product_id FROM `jos_csvi_icecat_index` AS `i` LEFT JOIN `jos_csvi_icecat_suppliers` AS `s` ON `s`.`supplier_id` = `i`.`supplier_id` WHERE `i`.`prod_id` = 'ITGCADCDSB' AND `s`.`supplier_name` = 'ITEK'

You said:

No I think this field is important

RO CSVI does not use this field.

What I wonder is, if you run the SQL query, in PhpMyAdmin for example, for the second product as stated above, do you get any result?

"What I wonder is, if you run the SQL query, in PhpMyAdmin for example, for the second product as stated above, do you get any result?"

I don't know how to run a query from PhpMyAdmin.
I've never done it. If you write me some instructions I can try.

Thanks

Hello,

Instead of running the query I have something better I think. I created an export template that outputs the product ID of the product you are trying to import if it exists. Can you restore the attached template via the Maintenance menu -> RO CSVI -> Restore template. After that run the Update available fields from the available fields page. Once that is done, you should be able to run the template and we can see what the result is.

Let me know how it goes.

Dear Roland
I did as you instructed and the result of your model is an error.
"Unfortunately, there are no fields set for this model. Now I don't know what to export. Go to models->model fields and add the fields you want to export."
I attach the log file and screenshot

I did the test with the ITGCACOLVB code and this gives the correct result.

Thanks

Hello,

After you ran the update list of available fields, please add the product_id field to your template and then run the export. This is a bit of a catch-22. RO CSVI does not restore the field because it does not exist but to be able to exist you need that template

Once that field is added the export should run.

Let me know how it goes.

I did the test and the result is this:
"No data was found for the selections made"

So it does not find the product_Id, right?
But if it is present in the ICECAT.index.csv file!!!
Can it be solved?

Thanks

Hello,

You are correct, this means that entry is not in your database. Yes, you are correct that it is in the index.csv file. So I think there must have been a hiccup importing that file. What you can do is reload the ICEcat index file again and then see if the entry is there.

Let me know how it goes.

Dear Roland

I had some problems that I will now describe to you.
I cleaned the temporary files folder from RO CSVI by deleting the old.index file.
I proceed with the upload of the icecat index files. Once the work is finished I see that the index file has a smaller size than usual. I repeat the operation and the work ends with a different size but not the expected one. This is a problem

So I write the index file from Icecat to my PC, I rename the file and insert it into the /tmp/com_csvi/ folder and start the upload of the icecat data. Once the work is finished I do the test with the model, it is fine. I start my test model with two products and check the log file and it is fine. I start my model and check the products of the "Itek" brand and they have photos and descriptions, it is fine

So it could have been a problem of corrupted data from the passage from the icecat_index file to the database server?
Now it seems to be going well and in the next few days I will keep an eye on the work.

I add another problem in this post
I see that I have many products without photos and descriptions because they have the MPN not written correctly (the data is provided by a supplier) or it does not exist. Can I ask you to add the product_gtin field in the product_Id search as an alternative to the product_mpn

can we continue the discussion in this place or should I open another one?

Thanks

Hello,

So I write the index file from Icecat to my PC, I rename the file and insert it into the /tmp/com_csvi/ folder and start the upload of the icecat data. Once the work is finished I do the test with the model, it is fine. I start my test model with two products and check the log file and it is fine. I start my model and check the products of the "Itek" brand and they have photos and descriptions, it is fine

Yes, saving the file to local PC and then uploading to /tmp/com_csvi/ folder is the way to solve if you have problem in downloading file index file from ICECat.

Glad to know your issue is solved.

Can I ask you to add the product_gtin field in the product_Id search as an alternative to the product_mpn

In VirtueMart Product Import template on Options tab set Import based on field to product_gtin and run the import. That should solve the issue.

Dear tharuna
Yes I have already tried this solution but it doesn't work.
Roland wrote that the search for the product_id in the database only happens with product_mpn, if I understood correctly.
From the test model provided by roland and from my other tests it doesn't work.

Thanks

Lorenzo

Hello Lorenzo,

Yes I have already tried this solution but it doesn't work.

Would like to see what does not work. Please post the latest import debug log file and import file to check.

Roland wrote that the search for the product_id in the database only happens with product_mpn, if I understood correctly.

I don't think he said it only works with product_mpn field. ICECat query uses two fields to find product_id, the field set in Import based on field on Options tab in import template and other is supplier name. Are you sure supplier name is right for the product_gtin you use for finding product_id?

If you post the import debug log I can find what happens in using product_gtin field.

Dear Tharuna
As I wrote you before, using product-gtin does not work at all.
Maybe the query used to find product_it is not correct.
I tested with this product from EZVIZ ean: 6941545607450.
I used the query from the ICEcat text export template provided by Roland.
I changed the line where 'prod_id' was and changed the 'ean-upc' field

SELECT product_id FROM `#__csvi_icecat_index` AS `i`
LEFT JOIN `#__csvi_icecat_suppliers` AS `s` ON ​​`s`.`supplier_id` = `i`.`supplier_id`
WHERE `i`.`ean_upc` = '6941545607450' AND `s`.`supplier_name` = 'Ezviz'

My model is set to search for product_gtin
the search response in my DB is positive product_id: 108686652

My model is set to search for product_gtin
the lines of the log file query shows me that it does not search for ean_upc
As you can see below the search is with EAN but the query is made to find prod_id.

Found ICEcat mpn reference: 6941545607450
2024-09-26 16:01:58 284 [DEBUG] Found ICEcat manufacturer name: EZVIZ
2024-09-26 16:01:58 284 [DEBUG] Find the ICEcat ID for manufacturer EZVIZ and part number 6941545607450
2024-09-26 16:01:58 284 [QUERY] SELECT product_id FROM `jos_csvi_icecat_index` AS `i` LEFT JOIN `jos_csvi_icecat_suppliers` AS `s` ON `s`.`supplier_id` = `i`.`supplier_id` WHERE `i`.`prod_id` = '6941545607450' AND `s`.`supplier_name` = 'EZVIZ'
2024-09-26 16:01:58 284 [DEBUG] Find the ICEcat ID by similar SKU for manufacturer EZVIZ and part number 6941545607450
2024-09-26 16:01:58 284 [DEBUG] Find the ICEcat ID as alternative ID for manufacturer EZVIZ and part number 6941545607450
2024-09-26 16:01:58 284 [QUERY] SELECT product_id FROM `jos_csvi_icecat_index` AS `i` LEFT JOIN `jos_csvi_icecat_suppliers` AS `s` ON `s`.`supplier_id` = `i`.`supplier_id` WHERE `i`.`m_prod_id` = '6941545607450' AND `s`.`supplier_name` = 'EZVIZ'
2024-09-26 16:01:58 284 [DEBUG] Received ICEcat data: []

It seems that the search fields prod_id and m-Prod_id are not correct

I attach the entire log file

Best regards

Hello Lorenzo,
There is no log file attached in your reply. Please post the import debug log so we can check.

I attach the entire log file

Hello Lorenzo,
Thank you for the debug log. We have made changes to ICECat code to use ean_upc value in search query. Please load the attached patch file and run the import. Let us know on how it goes.