Categories
View Replies (16)

Resolved Problem importing data from ICECAT

  1. fatelettronica
    fatelettronica
  2. RO CSVI
  3. Saturday, 31 August 2024
  4.  Subscribe via email
I noticed that sometimes analyzing the log file I find a problem with importing data from ICECAT.
I am sure that the data exists and can be downloaded for the "iTek" brand. in icecat I see that there is a photo, for example.
I write you the first lines of the log file where I find the anomaly.

2024-08-31 10:32:55 50 [DEBUG] Found ICEcat mpn reference: ITGCADCDSB
2024-08-31 10:32:55 50 [DEBUG] Found ICEcat manufacturer name: ITEK
2024-08-31 10:32:55 50 [DEBUG] Find the ICEcat ID for manufacturer ITEK and part number ITGCADCDSB
2024-08-31 10:32:55 50 [QUERY] SELECT product_id FROM `jos_csvi_icecat_index` AS `i` LEFT JOIN `jos_csvi_icecat_suppliers` AS `s` ON ​​`s`.`supplier_id` = `i`.`supplier_id` WHERE `i`.`prod_id` = 'ITGCADCDSB' AND `s`.`supplier_name` = 'ITEK'
2024-08-31 10:32:55 50 [DEBUG] Find the ICEcat ID by similar SKU for manufacturer ITEK and part number ITGCADCDSB
2024-08-31 10:32:55 50 [DEBUG] Find the ICEcat ID as alternative ID for manufacturer ITEK and part number ITGCADCDSB
2024-08-31 10:32:55 50 [QUERY] SELECT product_id FROM `jos_csvi_icecat_index` AS `i` LEFT JOIN `jos_csvi_icecat_suppliers` AS `s` ON ​​`s`.`supplier_id` = `i`.`supplier_id` WHERE `i`.`m_prod_id` = 'ITGCADCDSB' AND `s`.`supplier_name` = 'ITEK'
2024-08-31 10:32:55 50 [DEBUG] Received ICEcat data: []

As you can see it does not receive anything [ ]

While a correct answer, from another article, is this:
2024-08-31 10:32:02 1 [DEBUG] Calling ICEcat URL: https://data.icecat.biz/export/freexml.int/IT/87646936.xml
2024-08-31 10:32:02 1 [DEBUG] Dialling...
2024-08-31 10:32:02 1 [DEBUG] Finished talking to ICEcat
2024-08-31 10:32:02 1 [DEBUG] Closed connection

Thanks
0
Responses (16)
rolandd
rolandd
Accepted Answer Pending Moderation
Hello,

We have been through this before if I remember correctly. First thing to check would be, what happens if you import that product alone? Does it also not receive any data? The one thing that comes to mind is that there are too many requests from your server to ICEcat and that ICEcat stops responding for a moment.

Do that 1 product import and let's see how that goes.
Kind regards,

RolandD

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 1
fatelettronica
fatelettronica
Accepted Answer Pending Moderation
Dear Roland
I did as you asked.
I only put two products, both present in the ICECAT file "files.index.csv". One is ok but the one indicated in the previous post does not download the data. (Received ICEcat data: [])
There was no active cron job and nothing that can work with my and ICECAT's server
I attach the log file

Thanks
Attachments (1)
com_csvi.log.17.zip
  1. more than a month ago
  2. RO CSVI
  3. # 2
fatelettronica
fatelettronica
Accepted Answer Pending Moderation
Dear

If it can be useful I attach the two lines of the products in question, extracted from the file "files.index.csv"
Attachments (1)
righe_Index.txt
  1. more than a month ago
  2. RO CSVI
  3. # 3
rolandd
rolandd
Accepted Answer Pending Moderation
Hello,

Thank you for the provided information. So I had a look at the debug log you provided and decided to create my own import file.

So I created a super simple import file with just these lines and set the template to use file for configuration:


product_sku,manufacturer_name,product_name

ITGCADCDSB,ITEK,Test
That loads the ICEcat data as expected.
Now I had a further look at the log and we can see these lines:

  • Find the ICEcat ID for manufacturer ITEK and part number ITGCADCDSB
  • Find the ICEcat ID by similar SKU for manufacturer ITEK and part number ITGCADCDSB
  • Find the ICEcat ID as alternative ID for manufacturer ITEK and part number ITGCADCDSB
This means that RO CSVI cannot find the ICEcat URL. This is why you get no data back. One thing we need to check is if your supplier file has a supplier name ITEK and what is the supplier_id? In my case and in the file you provided this ID is 5237. Now I think that your supplier list has a different ID for ITEK, can you check if that is the case?

That is the only reason I can think of why it would not load the data or that the ITEK supplier does not exist.

Let me know what you find.
Kind regards,

RolandD

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 4
fatelettronica
fatelettronica
Accepted Answer Pending Moderation
Dear Roland

I think you didn't see the file I sent you "righe_index.txt"
Inside the file there are two lines copied from the file "file.index.csv"
The two lines correspond to the products used for the test.
In the file there is the product ITGCACOLVB manufacturer ITEK (Line 1) and this has no problems downloading the data from icecat.
Then there is the product ITGCADCDSB manufacturer ITEK (Line 2) this is the one that has problems.
As you will see both have the supplier_id = 5237 = ITEK.

There is a field, on_market (field to the right of the EAN/UTC code field), which I don't know what it's for, I see that there is a 1 in the ok product while it is 0 in the product with the problem. No I think this field is important

Thanks
Attachments (1)
righe_Index.txt
  1. more than a month ago
  2. RO CSVI
  3. # 5
rolandd
rolandd
Accepted Answer Pending Moderation
Hello,

I definitely saw the righe_index.txt file, as I still have it open on my desktop :) As I mentioned before, the log shows that RO CSVI cannot find a reference to your second product, that is why no data is available. RO CSVI does not even try to contact ICEcat because it has no URL.

For the first product it immediately finds a reference so it can call ICEcat. That is why your log only shows this:
Find the ICEcat ID for manufacturer ITEK and part number ITGCACOLVB
while for the second product it tries 3 times to find a corresponding URL but finds none.

The first product is found using this query:


SELECT product_id FROM `jos_csvi_icecat_index` AS `i` LEFT JOIN `jos_csvi_icecat_suppliers` AS `s` ON `s`.`supplier_id` = `i`.`supplier_id` WHERE `i`.`prod_id` = 'ITGCACOLVB' AND `s`.`supplier_name` = 'ITEK'
RO CSVI is trying to find the second product using this query 


SELECT product_id FROM `jos_csvi_icecat_index` AS `i` LEFT JOIN `jos_csvi_icecat_suppliers` AS `s` ON `s`.`supplier_id` = `i`.`supplier_id` WHERE `i`.`prod_id` = 'ITGCADCDSB' AND `s`.`supplier_name` = 'ITEK'
You said:
No I think this field is important
RO CSVI does not use this field.

What I wonder is, if you run the SQL query, in PhpMyAdmin for example, for the second product as stated above, do you get any result?
Kind regards,

RolandD

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 6
fatelettronica
fatelettronica
Accepted Answer Pending Moderation
"What I wonder is, if you run the SQL query, in PhpMyAdmin for example, for the second product as stated above, do you get any result?"

I don't know how to run a query from PhpMyAdmin.
I've never done it. If you write me some instructions I can try.

Thanks
  1. more than a month ago
  2. RO CSVI
  3. # 7
rolandd
rolandd
Accepted Answer Pending Moderation
Hello,

Instead of running the query I have something better I think. I created an export template that outputs the product ID of the product you are trying to import if it exists. Can you restore the attached template via the Maintenance menu -> RO CSVI -> Restore template. After that run the Update available fields from the available fields page. Once that is done, you should be able to run the template and we can see what the result is.

Let me know how it goes.
Attachments (1)
csvi_templates_csvi.test_20240904.xml
Kind regards,

RolandD

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 8
fatelettronica
fatelettronica
Accepted Answer Pending Moderation
Dear Roland
I did as you instructed and the result of your model is an error.
"Unfortunately, there are no fields set for this model. Now I don't know what to export. Go to models->model fields and add the fields you want to export."
I attach the log file and screenshot

I did the test with the ITGCACOLVB code and this gives the correct result.

Thanks
Attachments (2)
com_csvi.log.26.zip
Error_Test.pdf
  1. more than a month ago
  2. RO CSVI
  3. # 9
rolandd
rolandd
Accepted Answer Pending Moderation
Hello,

After you ran the update list of available fields, please add the product_id field to your template and then run the export. This is a bit of a catch-22. RO CSVI does not restore the field because it does not exist but to be able to exist you need that template :)

Once that field is added the export should run.

Let me know how it goes.
Kind regards,

RolandD

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 10
fatelettronica
fatelettronica
Accepted Answer Pending Moderation
I did the test and the result is this:
"No data was found for the selections made"

So it does not find the product_Id, right?
But if it is present in the ICECAT.index.csv file!!!
Can it be solved?

Thanks
  1. more than a month ago
  2. RO CSVI
  3. # 11
rolandd
rolandd
Accepted Answer Pending Moderation
Hello,

You are correct, this means that entry is not in your database. Yes, you are correct that it is in the index.csv file. So I think there must have been a hiccup importing that file. What you can do is reload the ICEcat index file again and then see if the entry is there.

Let me know how it goes.
Kind regards,

RolandD

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 12
fatelettronica
fatelettronica
Accepted Answer Pending Moderation
Dear Roland

I had some problems that I will now describe to you.
I cleaned the temporary files folder from RO CSVI by deleting the old.index file.
I proceed with the upload of the icecat index files. Once the work is finished I see that the index file has a smaller size than usual. I repeat the operation and the work ends with a different size but not the expected one. This is a problem

So I write the index file from Icecat to my PC, I rename the file and insert it into the /tmp/com_csvi/ folder and start the upload of the icecat data. Once the work is finished I do the test with the model, it is fine. I start my test model with two products and check the log file and it is fine. I start my model and check the products of the "Itek" brand and they have photos and descriptions, it is fine

So it could have been a problem of corrupted data from the passage from the icecat_index file to the database server?
Now it seems to be going well and in the next few days I will keep an eye on the work.

I add another problem in this post
I see that I have many products without photos and descriptions because they have the MPN not written correctly (the data is provided by a supplier) or it does not exist. Can I ask you to add the product_gtin field in the product_Id search as an alternative to the product_mpn

can we continue the discussion in this place or should I open another one?

Thanks
  1. more than a month ago
  2. RO CSVI
  3. # 13
tharuna
tharuna
Accepted Answer Pending Moderation
Hello,
So I write the index file from Icecat to my PC, I rename the file and insert it into the /tmp/com_csvi/ folder and start the upload of the icecat data. Once the work is finished I do the test with the model, it is fine. I start my test model with two products and check the log file and it is fine. I start my model and check the products of the "Itek" brand and they have photos and descriptions, it is fine

Yes, saving the file to local PC and then uploading to /tmp/com_csvi/ folder is the way to solve if you have problem in downloading file index file from ICECat.

Glad to know your issue is solved.


Can I ask you to add the product_gtin field in the product_Id search as an alternative to the product_mpn

In VirtueMart Product Import template on Options tab set Import based on field to product_gtin and run the import. That should solve the issue.
Kind regards,

Tharuna

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 14
fatelettronica
fatelettronica
Accepted Answer Pending Moderation
Dear tharuna
Yes I have already tried this solution but it doesn't work.
Roland wrote that the search for the product_id in the database only happens with product_mpn, if I understood correctly.
From the test model provided by roland and from my other tests it doesn't work.

Thanks

Lorenzo
  1. more than a month ago
  2. RO CSVI
  3. # 15
tharuna
tharuna
Accepted Answer Pending Moderation
Hello Lorenzo,
Yes I have already tried this solution but it doesn't work.

Would like to see what does not work. Please post the latest import debug log file and import file to check.

Roland wrote that the search for the product_id in the database only happens with product_mpn, if I understood correctly.

I don't think he said it only works with product_mpn field. ICECat query uses two fields to find product_id, the field set in Import based on field on Options tab in import template and other is supplier name. Are you sure supplier name is right for the product_gtin you use for finding product_id?

If you post the import debug log I can find what happens in using product_gtin field.
Kind regards,

Tharuna

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 16
  • Page :
  • 1


There are no replies made for this post yet.
Be one of the first to reply to this post!