Categories
View Replies (41)

Resolved Virtuemart Prices import - Huge problem

  1. roderic
    roderic
  2. RO CSVI
  3. Sunday, 16 January 2022
  4.  Subscribe via email
Hello,

I have 5 templates (attached) to import prices and discounted prices. These are imported every night and usually works like expected. There have been a few times, and it seems to occur more often recently, that the discounted price of product X also gets added to Product Y. The product that gets the price from the other product, is one row below in the XML file. This has lead to 2 orders of products with a price of around 100 euro for a product that's actually about 6000 euro and one that was 1700 euro. This of course is a big issue. As soon as we've seen problems like this, I've manually run the 5 templates and then the price was normal again.

Today however I also had this problem after manually importing and thus I have the corresponding log file. I've attached this logfile (log.1703) and if you look for:
virtuemart_product_id` = 366

You will find the product that got the wrong price (101.2500). This price belongs to virtuemart_product_id` = 5896

As you can see in the logfile, this product is being dealt with just before the other one. The product with ID 366 should be skipped with this template. That's what happened when I ran the same import again.

Could it be that sometimes the import is still working on one product and it already starts with the next product but then these values get mixed up?

The XML file is also attached.
Attachments (3)
com_csvi.log.1703.rar
csvi_templates_www.woest-sport.nl_20220116.xml
Products.rar
0
Responses (41)
tharuna
tharuna
Accepted Answer Pending Moderation
Hello Roderic,
The imports of Variants do seem to take a bit longer. Maybe because of the additional logging?

From the debug log i see import has taken 5 mins and that is only for the records in debug log. Not sure if there are more records which are not in debug log. So checking the time taken for each record i see they are in few milliseconds but the number of records imported are more. Probably you imported a large file?
Kind regards,

Tharuna

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 1
roderic
roderic
Accepted Answer Pending Moderation
Hi Tharuna,

Unfortunately I only now had the chance to test the patch. So far so good. I've attached the logfile.

No, i haven't installed the patch on test site as i don't have access to it anymore. Administrator link takes me to frontend of the site. Please install the patch file and run the import. Let me know on how that goes.


There was a security measure from Akeeba Admin tools, that blocks admin logins during the night. This has been there all the time but I guess you've always logged in after this period. I've now removed it for the test website.

The imports of Variants do seem to take a bit longer. Maybe because of the additional logging?

I need to run now but I can do a few more test runs later on.

Cheers
Attachments (1)
com_csvi.log.252.zip
  1. more than a month ago
  2. RO CSVI
  3. # 2
tharuna
tharuna
Accepted Answer Pending Moderation
Hello Roderic,
Did you get a chance to test the patch? Please let us know on how the import go.
Kind regards,

Tharuna

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 3
tharuna
tharuna
Accepted Answer Pending Moderation
Hello Roderic,
That sounds promising!

Not so soon :D

Have you also installed this on the test site? If not, I'll install it and do some testing there.

No, i haven't installed the patch on test site as i don't have access to it anymore. Administrator link takes me to frontend of the site. Please install the patch file and run the import. Let me know on how that goes.
Kind regards,

Tharuna

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 4
roderic
roderic
Accepted Answer Pending Moderation
Hi Tharuna,

That sounds promising!

Have you also installed this on the test site? If not, I'll install it and do some testing there.

Thanks!
  1. more than a month ago
  2. RO CSVI
  3. # 5
tharuna
tharuna
Accepted Answer Pending Moderation
Hello Roderic,
Because I have set Delete custom fields set to No, so importing a product halfway the import will not interfere with products at the start of the import

Yes understood the difference :)

I could run a test where I truncate the table first on the test site, then import with Delete custom fields set to Yes.

I think i may have found the cause for the issue of values not linked after the import. I have fixed it with the attached patch file. I have also added few more logging so to track the import process. Run import with attached patch file and if you still see the issue, please post the latest debug log to check.
Attachments (1)
patch_cf4all_2022_02_24.zip
Kind regards,

Tharuna

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 6
roderic
roderic
Accepted Answer Pending Moderation
Hi Tharuna,

Won't it empty the values of products which are not in import file?


Yes it will, but all products are imported every night, including customfields.

Delete custom fields set to Yes, deletes the values for products in import file. I am wondering how truncating the table does not create an issue but delete custom fields is a problem.


Because I have set Delete custom fields set to No, so importing a product halfway the import will not interfere with products at the start of the import

I could run a test where I truncate the table first on the test site, then import with Delete custom fields set to Yes.
  1. more than a month ago
  2. RO CSVI
  3. # 7
tharuna
tharuna
Accepted Answer Pending Moderation
Hello Roderic,
It happens with many products, but it's quite random.

Unless we can reproduce the issue, it is difficult to fix it.

As I've mentioned before, I'm running the import every night now with first truncating the customfields table. This so far gives reliable/consistent results. The setting in the template for "delete custom fields" is then set to no (logically as the table is empty anyway after truncating)

Won't it empty the values of products which are not in import file? Delete custom fields set to Yes, deletes the values for products in import file. I am wondering how truncating the table does not create an issue but delete custom fields is a problem.
Kind regards,

Tharuna

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 8
roderic
roderic
Accepted Answer Pending Moderation
Hi Tharuna,

I guess we are going in circles with this issue.


Exactly my thoughts.

Now i am confused if this product has the issue of values being removed or any other product


It happens with many products, but it's quite random. As I've mentioned before, I'm running the import every night now with first truncating the customfields table. This so far gives reliable/consistent results. The setting in the template for "delete custom fields" is then set to no (logically as the table is empty anyway after truncating)

gets all the values when Delete custom fields set to Yes


Yes it does, though it sometimes first removes certain values, then reimports them. Then for this products all is fine but not so much for the products that came before this product in the import because the values are then not linked anymore.


Cheers
  1. more than a month ago
  2. RO CSVI
  3. # 9
tharuna
tharuna
Accepted Answer Pending Moderation
Hello Roderic,
I guess we are going in circles with this issue.

The product you are saying there is an issue in the linked topic, gets all the values when Delete custom fields set to Yes. I have been using that product as reference in checking the issue and have run the import quite a few times on your site. Now i am confused if this product has the issue of values being removed or any other product :o
Kind regards,

Tharuna

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 10
roderic
roderic
Accepted Answer Pending Moderation
Hello Roderic,
May i know why not set Delete custom fields to Yes with child product import template(ID 77)?


Hi Tharuna,

As mentioned before in this topic, setting it to YES will then cause the problem of sometimes a value being removed that was already linked to another product:

https://rolandd.com/support/ro-csvi/virtuemart-prices-import-huge-problem#reply-70191
  1. more than a month ago
  2. RO CSVI
  3. # 11
tharuna
tharuna
Accepted Answer Pending Moderation
Hello Roderic,
Yes, i changed those import files and ran the import. I see that Delete custom fields was set to yes in template with ID 74 and with template ID 77, Delete custom fields was set to No. This was creating number of rows for child product in #__virtuemart_product_customfields table. That is the reason why values are not selected with Size custom field. I set Delete custom fields to Yes in template ID 77 and ran the import. This solved the issue of values not selected. May i know why not set Delete custom fields to Yes with child product import template(ID 77)?
Kind regards,

Tharuna

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 12
roderic
roderic
Accepted Answer Pending Moderation
Hi Tharuna,

Am i missing something?


No, if you change the imported filename for templates 74+77 to Products-test.xml instead of Products.xml, you will be able to import the Stockablefields (74) and then they will be filled with values. But only because Products-test.xml contains only that product. When importing the complete file, it fails.

Cheers
  1. more than a month ago
  2. RO CSVI
  3. # 13
tharuna
tharuna
Accepted Answer Pending Moderation
Hello Roderic,
Yes, import went without any problem. Thank you.

So i ran import with three templates as you mentioned and see no stockablecustomfields values added for product 110BA760-CDCA-4FE5-A809-99FBC26ABD5C (the product from your screenshot). Am i missing something?
Kind regards,

Tharuna

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 14
roderic
roderic
Accepted Answer Pending Moderation
Hi Tharuna,

The problem is that category_id is getting product_sku value for one product on import which is wrong. Any idea why only this product gets that value for category_id?


It's the category ID. There is a conditional rule that changes every Vendit ID to a Virtuemart ID. But since this is a new category (added yesterday), there was no rule for that specific category yet on the test site. There is now and thus the error is gone and the import is able to finish.

Sorry about that
  1. more than a month ago
  2. RO CSVI
  3. # 15
tharuna
tharuna
Accepted Answer Pending Moderation
Hello Roderic,
What I meant was, that one of the last patches you've given me, fixed one thing but also had older code in there for another problem that was fixed before. Because basically a patch file overwrites a complete file, right?

Yes, patch overwrites complete file.

In the meantime I've added a cronjob that truncates virtuemart_custom_plg_customsforall & virtuemart_product_customfields every night just before the overnight imports. Than I can leave delete customfields in the templates to no and then at least I can make sure everything is imported for now. Running a manual import during the day, in case needed, is then a problem.

I have updated test site with latest RO CSVI package to test the import but have been running into errors. There is a duplicate error with category_id. See attached screenshots. The problem is that category_id is getting product_sku value for one product on import which is wrong. Any idea why only this product gets that value for category_id?

I've disabled the triggered rule (218140) and now you should be able to continue.

Yes, there is no 403 forbidden error now. Thank you.
Attachments (2)
Screenshot 2022-02-15 at 5.17.45 PM.png Screenshot 2022-02-15 at 5.17.45 PM.png
Screenshot 2022-02-15 at 5.20.06 PM.png Screenshot 2022-02-15 at 5.20.06 PM.png
Kind regards,

Tharuna

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 16
roderic
roderic
Accepted Answer Pending Moderation
Hello Tharuna,

you have installed an outdated patch


What I meant was, that one of the last patches you've given me, fixed one thing but also had older code in there for another problem that was fixed before. Because basically a patch file overwrites a complete file, right?

In the meantime I've added a cronjob that truncates virtuemart_custom_plg_customsforall & virtuemart_product_customfields every night just before the overnight imports. Than I can leave delete customfields in the templates to no and then at least I can make sure everything is imported for now. Running a manual import during the day, in case needed, is then a problem.


May i know what is causing this access forbidden error?


Modsecurity:



[client 49.205.123.xxx] ModSecurity: [file "/etc/httpd/conf/modsecurity.d/rules/comodo_free/17_Outgoing_FilterSQL.conf"] [line "91"] [id "218140"] [rev "1"] [msg "COMODO WAF: mysql SQL Information Leakage|||F|2"] [data "Matched Data: error found within MATCHED_VAR: <!DOCTYPE html>\\x0a<html lang=\\x22en-gb\\x22 dir=\\x22ltr\\x22>\\x0a<head>\\x0a\\x09<meta name=\\x22viewport\\x22 content=\\x22width=device-width, initial-scale=1.0\\x22>\\x0a\\x09<meta http-equiv=\\x22X-UA-Compatible\\x22 content=\\x22IE=edge\\x22 />\\x0a\\x09<meta charset=\\x22utf-8\\x22 />\\x0a\\x09<meta name=\\x22keywords\\x22 content=\\x22WOEST, sport, running, bikes, triathlon\\x22 />\\x0a\\x09<meta name=\\x22description\\x22 content=\\x22WOEST Sport voor alles op het gebied van bike..."] [severity "CRITICAL"] [tag "CWAF"] [tag "FilterSQL"] Access denied with code 403 (phase 4). Pattern match "(?i)(?:supplied argument is not a valid MySQL|Column count doesn't match value count at row|mysql_fetch_array\\\\(\\\\)|on MySQL result index|You have an error in your SQL syntax;|You have an error in y [hostname "woest-sport.nl"] [uri "administrator/index.php"] [unique_id "YgotsDiKwM5olOktmYyk@wAAAEc"], referer: administrator/index.php?option=com_csvi&task=importsource.source&runId=428



I've disabled the triggered rule (218140) and now you should be able to continue.
  1. more than a month ago
  2. RO CSVI
  3. # 17
tharuna
tharuna
Accepted Answer Pending Moderation
Hello Roderic,
I was testing the Stockable custom fields import on the test site and see that not just one but none of the Size values are linked to the child products. I installed latest RO CSVI Package on the test site so to be sure all the fixes are included. Now when i run the import, i see 403 forbidden error with import preview page. See attached screenshot. May i know what is causing this access forbidden error?
Attachments (1)
Screenshot 2022-02-14 at 3.53.50 PM.png Screenshot 2022-02-14 at 3.53.50 PM.png
Kind regards,

Tharuna

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 18
tharuna
tharuna
Accepted Answer Pending Moderation
Hello Roderic,
It is really bad again now. It is almost as if one of the patches I recently installed undid one of the earlier patches. Could this be possible?

Yes, it is possible that you have installed an outdated patch which would have overridden the latest code changes. Can you tell me what is broken?

I will look into the stockable custom fields issue and send you an update.
Kind regards,

Tharuna

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 19
roderic
roderic
Accepted Answer Pending Moderation
Hi again,

It is really bad again now. It is almost as if one of the patches I recently installed undid one of the earlier patches. Could this be possible?
  1. more than a month ago
  2. RO CSVI
  3. # 20
  • Page :
  • 1
  • 2
  • 3


There are no replies made for this post yet.
Be one of the first to reply to this post!