XML Import from Vendit into Virtuemart #65994

Hello,

XML attached.

"Also tell us how are you expecting subgroups categories to be read? Are they subcategories of Bikes category or they are individual categories?" They can be subcategories of several parent categories.

"Can you give an example? "
The subgroup/subcategory in the previously attached groups.xml, Onderdelen is a subcategory of Bikes
As can be seen by:

<Parent_Guid>898714B0-BB0C-4541-A325-D967B9EC43B1</Parent_Guid>

which is the same Guid as

<GroupGuid>898714B0-BB0C-4541-A325-D967B9EC43B1</GroupGuid>

From Bikes

The problem seems to be that CSVI just stops after the parent category because it get's confused by the

<SubGroups>

tag

Cheers

• csvi_templates_www.woest-sport.nl_20210226.xml

Hello,
Ok so the category_path to be built will be Bikes/Onderdelen/Aandrijving, is that correct?

Correctomundo

The problem is not with SubGroups tag but Group tab after SubGroups. RO CSVI reads the XML based on the record name set in template settings. In the Record name you have set it to Group and the node name after SubGroups is also Group. That is the problem here.

What would be a solution to this, without messing with the generated XML file from VendIT?

Thanks for your patch file. I tried playing around with it for a while. This should then also work with Products right?

Could you please give me one or two more examples with the newly attached file for:

price_with_tax: Products/Product/ProductVariations/ProductVariation/SalesPriceInc
and
product_gtin: Products/Product/ProductVariations/ProductVariation/Barcodes/Barcode BarcodeOrder="1"

Thanks in advance

• Products(1).xml

Hello again,

I've been working on this quite a bit, but haven't found the time yet to reply. So here we go:

First of all, the RO CSVI Analyser didn't work because of the UTF-16 encoding but today (I'm at my client) I've managed to get exports with UTF-8 encoding so that helps with getting more insight.

I have a problem with the categories/groups. When I import the Groups.xml, it only takes the first (sub)Group/Category and then goes to the next.

So for example:

• 0 Bikes
  -- 0 Fietsen
  --- 0 Racefietsen
  --- 1 Gravelbikes
  --- 2 Mountainbikes
  --- 3 Triathlonfietsen
  -- 1 Fietskleding
  -- 2 Onderdelen
  -- 3 Accesoires

It will see all the (sub)Groups) wich are the first subgroups (those in bold), but not the 2nd, 3rd etc.

Does this require another patch?

• Products.xml
• Groups.xml

Hello,

First of all, the RO CSVI Analyser didn't work because of the UTF-16 encoding but today (I'm at my client) I've managed to get exports with UTF-8 encoding so that helps with getting more insight.

RO CSVI Analyser would have shown unreadable characters for files other than UTF-8 encoded. Is that what you are saying not working? Can you post a screenshot on what you see?

It didn't do anything, I would select the file, uncheck "has column headers" and fill in a record name, then hit analyse but then it would just empty the fields and not give any output. But now with the UTF8 files it works as expected.

I have a problem with the categories/groups. When I import the Groups.xml, it only takes the first (sub)Group/Category and then goes to the next.

Please post the template XML to check. To get your template XML file, in RO CSVI maintenance menu select RO CSVI on first option, backup templates on second option, select your template and download the XML file. Post the XML file. With this template and your file, i can run the import and check what is going on with categories.[/quote]

XML attached

• 0 Bikes
  -- 0 Fietsen
  --- 0 Racefietsen
  --- 1 Gravelbikes
  --- 2 Mountainbikes
  --- 3 Triathlonfietsen
  -- 1 Fietskleding
  -- 2 Onderdelen
  -- 3 Accesoires

It will see all the (sub)Groups) wich are the first subgroups (those in bold), but not the 2nd, 3rd etc.

Can you give an example on how you expect category to be built with these values? Also tell me on how it is built right now with your import template.
[/quote]

So I expect it to import all the categories and subcategories. So the proper hierachy:
Bikes / Fietsen / Racefietsen (works right now)
and
Bikes / Fietsen / Gravelbikes (doesn't work right now as it is the not the first subGroup of a Subgroup)

But also

Bikes / Fietskleding / Fietsjacks

and

Running / Hardloopschoenen / Allround hardloopschoenen

Basicallly all the Subgroups with <ItemOrder>0</ItemOrder> are being imported and <ItemOrder>1</ItemOrder> or more are not.

• csvi_templates_www.woest-sport.nl_20210331.xml

I basically want all the categories / category paths that are in the Groups.xml file imported into Virtuemart. So every parent and child category.

See attachment for an example. Running is imported like expected except that it doesn't import the 2nd subgroup/childcategories in every category like I've explained before. So in my example the Category Running, Hardloopschoenen and Allround Hardloopschoenen are imported with the current import settings. The other 2 on my screenshot, wedstrijdschoenen and trailschoenen I had to add manually because they were ignored by the import.

Cheers

Thanks so far, anything I can do to speed this up?

Hi Roland,

Thanks for your reply. Right now it requires a category path to be able to have a proper hierachy. But when making use of the Parent group ID (<Parent_Guid>), this could maybe be make it easier?

Met vriendelijke groet,
Roderic

Hoi Ronald,

That's looking good. But in your screenshot I'm also missing:
Bikes/Fietsen/Mountainbikes,
Bikes/Fietsen/Baanfiets,
Bikes/Fietsen/Triathlonfietsen

Expected or overlooked?

Cheers,

Roderic

Hoi Roland,

In the meantime I've been working on importing/exporting other things from Vendit in trying to get this project to an end.

1. Category import

2. Parent Product Import

3. Child Products import

4. Stock import

5. VirtueMart Userinfo Import

6. VirtueMart Userinfo Export

7. Order export

8. For this I'm waiting for your update, any estimate?

9. I made a template to import the (parent) products, used combine rules to make sure all the images are imported. I also tried importing the custom fields and ran into a few problems there.
   

It looks good in this preview, but it will only import the first color out of 3 and put an "|" behind it. (I also tried to import it like #ffffff|#ffffff|#ffffff and #ffffff/#ffffff/#ffffff)

Also, the label is left empty, I don't think the Documentation covers this part.

3. I've duplicated the Product import and changed the node to

<Products><Product><ProductVariations><ProductVariation>

This will list all the 204 product variants currently in the Products.xml. But the problem is, there is no way to tell which child/variant belongs to which parent. So I guess that's not the way to go. I hope that the changes you are working on for Groups.xml, will also help me in getting (all) the other product import template. Or should I still import the parents first and make a separate template for child products? I want to make use of the breakdesigns Stockable + Customfields for all.

4. Works, I've set up a cronjob that updates the stock for the products that are currently imported for every few hours.

5. Works, I've set up a cronjob that imports new clients every day and they are also automatically imported to AcyMailing

6. Made a template with all the required fields by Vendit. One of them is:

  <ImportInfo>
     <ExportDateTime> </ExportDateTime>
  </ImportInfo>

In the XML Header. Is there a way to generate this date/time?

Also, would there be a way to limit the userinfo's getting exported to users that have placed an order in the last 2 days only?
Maybe a skip rule like this?

And then add that to last visit date or so..

7. Same as 6. Also, would it be possible to trigger this Order export when a client has placed an order instead of a cronjob? (Not that important)

Cheers

Good morning,

1. Sounds good

2. You can add labels here:
   
   In the database it then looks like:
   
   In the frontend it will show up in the module:
   

So it is used, although it's not the biggest deal when it's left empty.

3. Attached are the most recent products.xml and the most recent import template. Also the docs I got from Vendit (Could be helpful, maybe).

4. And how about this question? "In the XML Header. Is there a way to generate this date/time?" what code should I put there so it will always put the correct date there?

Cheers!

• Products(1).xml
• csvi_templates_www.woest-sport.nl_20210413.xml
• E-commerce 2.0 groepen.docx
• E-commerce 2.0 producten.docx
• E-commerce 2.0 order.docx

Hi Roland,

Thanks for your reply, I have played around with it and figured out it will only work in the Body part of the XML and I need to have it in the Header.
When I use the same tag in the header, it will just show the name of the tag instead.

So, when tag placed in header:

  <ImportInfo>
     <ExportDateTime>[datetime]</ExportDateTime>
  </ImportInfo>

When placed in Body:

<Customers>
<ExportDateTime2>2021-Apr-TueUTC02:04:05+01:00</ExportDateTime2>
<Customer>

Hello Roland,

Yes I am using the custom option.

While I'm waiting for your Patch or update for the Categories/Groups, I've tried to get the Customfields part ready of the Products import.
According to the documentation, I need to have a field with custom_multiple with the same amount of Y as there are child products.

Since there is no Y value in the XML file, I thought of a possible workaround. I combined another field called ProductVariations/ProductVariation/IsMandatoryExtra which always has the value False

Then it will show up like this:

Then I've used Conditional rule to replace the False with Y

And it will look like:

Is this the way to do it or am I overcomplicating it?

Hoi Roland,

Thanks for the patch, installed and it worked. Although it didn't listen to the time settings I've set in the rule but instead it listens the the Date format on the File tab. I've set it to Y-m-d\TH:i:sP and now it is correct.

No patch for the Categories yet?

I've now managed to import Stockable products + Parent's product's Size and Color. The problem now is that the ID of the 2nd/3rd etc. Stockable product get's the same ID as the parent. Meaning it will not help to make a child import. Also, I tried using skip in the child template so it will take the second child's Size and Colour. But as soon as I use the skip fields, the other rules stop working. For example, I've made a rule that makes a color code from text, these stop working when I use skip fields.

So I'm hoping that your fix for categories will also bring something for child products.

Basically all I would need now for the child products import, is an extra rule/field in the main product template that copies the ID's of the child products to the Stockable fields. Just like it does with color/size. I feel like I'm almost there but just a few small things stopping me from finalizing this..

This missing ID in Stockable:

What If those could be fetched from here:

I could then run a child template to add the Size and Color (or any other value) because I can then update the child products based on SKU/ID

Cheers!

Edit: added 2 images

Hello Tharuna,

1. Yes, I have 41 Parent products in the database and they all have stockable customfields.

So here's an example, a trimmed down version of Products.xml:

<Products>
<Product><ProductNumber>30000408</ProductNumber>
<PageTitle>BMC Timemachine 01 ROAD MOD</PageTitle>
<ProductVariations>
<ProductVariation><ProductId>115977-0</ProductId><Size>47</Size><Color>pet bru bru</Color></ProductVariation>
<ProductVariation><ProductId>115978-0</ProductId><Size>51</Size><Color>pet bru bru</Color></ProductVariation>
<ProductVariation><ProductId>115979-0</ProductId><Size>54</Size><Color>pet bru bru</Color></ProductVariation>
<ProductVariation><ProductId>115980-0</ProductId><Size>56</Size><Color>pet bru bru</Color></ProductVariation>
<ProductVariation><ProductId>115981-0</ProductId><Size>58</Size><Color>pet bru bru</Color></ProductVariation>
<ProductVariation><ProductId>115982-0</ProductId><Size>61</Size><Color>pet bru bru</Color></ProductVariation>
</Product>
</Products>

The first ProductVariation is the Parent SKU. In this example it's 115977-0 with size 47

The Parent product is imported with stockable customfields, meaning this Parent product get's 6 stockable customfields aka 1 parent+ 5 childproducts. The problem here is that these 6 stockable customfields have the same ID:

2. "The product_sku field in your screenshot has multiple values separated by ~, how do we know which are child product SKUs in those values?"

I did the import with just one value, the same value as the parent SKU. Just to show as an example, I made the combine fields for product_sku and only ran a "preview" so I could make a screenshot. I did then cancel the import.

3. "From the screenshots i think you want to read the product_sku value, find the product id based on it and update customfield_params with product id. Is that what you are trying to achieve? "

What I am trying to achieve is, that it adds the product_sku values of my child products to the stockable customfields, like in the screenshot above. When that would work, I could basically have all Parent and child products added with one template. I could then make a second import template, based on the product sku's, to update things like MPN/GTIN(EAN). Or to add the Size, if that's not possible with the main product import.

An example of stockablecustomfield import is explained in the document here

Yes I followed this exact document to try and get it to work. I did all the steps and added all the necessary information. Just the issue of not having the child products. But the way CSVI works, this basically only works with CSV files, right?

What would help out greatly, when you could override the XML Path node for just one or more fields with a rule. For example, right now I've set the node to Product and when importing it will see 41 products. What if I could overrule the node for just product_sku to be ProductVariation. That way you could have parent and child import in one go. Would this be technically possible? So a Path node override Rules plugin. I think that could solve most, if not all problems

Good evening,

Ok so I thought I found a smart workaround for parent/child. I used the Description field as product_sku as this field shows up in both <Product> as well as <ProductVariation>. I made a rule to remove the whitespace. So far so good.

I imported the parent products with these sku's. All good. Now I made a template for the child products with the same setup and node set to ProductVariation. I hit preview, it says 204 lines (204 child products). Now when I hit start, it will only import the first 6 lines (one product) and then just stop. I guess this is the same issue as you were still having with the Category/Group import @rolandd# ?

Also, I mentioned before, I was trying to import the colors for customfieldsforall. You either misunderstood me or I misunderstood you:

Colors are imported without the # for the color. The # is used to separate multiple values. To import the multiple colors for a product your field needs to look like ffffff#ffffff#ffffff. The | (pipe symbol) is added in the database for custom fields for all so they know there are multiple values.The customsforall_value_label field is currently not supported for import. As far as I can see this is not used for the colors.

Either way, In the end I removed the #, replaced them for pipes and made the import like: ffffff|000000|ffffff. This has the desired result:

Maybe something to add to the documentation for someone else who is also looking into getting more than one Color into that box?

Coming back to writing a custom rules plugin myself

you can create a new RO CSVI rule plugin and write custom code to read the SKU of child products, find the child product id, write query to get customfield_params field based on parent product ID and update child id to the field.

I've looked into it, looked at the php files of other plugins, but I'm afraid that's out of my scope. I guess I will have to wait for the update and see if that fixes both of my remaining problems.

Cheers

Good evening,

The import should not stop after 6 lines if it says 204. How many rows do you see in the analyzer with ProductVariation if you set the number of lines to preview to 10?

3, at the 3rd product it places everything in one long line. I just tested it at another website where I did not install the Patch I got from @tharuna# and there it lists all 10. Or even all 204 when I enter 204 to preview. Although then there seem to be a few double entries:

Can I temporarily disable a Patch?

Cheers

Hi Roland,

Thanks, I will give this a try later. But I already wanted to reply to this part:

One thing I noticed is that the products have no specific category information other than the ProductGroup GUID. RO CSVI cannot look up this information during import, so either you need to create a custom rule that does this or use the replacement rule but that requires having all the GUIDs and actual paths available.

Yesterday I added all the categories manually, then made the rule above so it knows which guid is which category. This worked for me for the products that I have imported multiple times. I will now see if the category import will recognize this and then add the description / meta stuff to the existing categories instead of creating new ones.

Also, last night I installed a new Joomla website with CSVI without any patches and then my plan quoted below did work out as expected/hoped.

Ok so I thought I found a smart workaround for parent/child. I used the Description field as product_sku as this field shows up in both <Product> as well as <ProductVariation>. I made a rule to remove the whitespace. So far so good.

Have a good weekend as well!

Hi Roland,

So I've been testing the patch thoroughly and the common theme seems to be, inconsistency.

With the way you have set it up for the categories/groups in your template, I only get 3 categories when I run the import. (Bikes / Fietsen / Racefietsen).
It seems like it won't go in to the next Group / Subgroup when you set the path that deep. Did you manage to import more than just 3 categories with your template?

Now for the Products I feel we've made some progress but at the same time some new/other problems.

When I set the path to:

1. ProductExport/Products/Product

Preview shows 40 products. It will not find the productId's of the different variations or Attributes/Attribute/Name for example.

For nodes with extra values between comments, it will read those instead of the actual value.

<Groups>
<ProductGroup Default="True">37878FEC-DF8C-45AD-A1D3-B0180D2AB7F6</ProductGroup>
</Groups>

<Barcodes>
<Barcode BarcodeOrder="1">7630027861693</Barcode>
</Barcodes>

So the ProductGroup will return True the Barcode will return a 1.

2. ProductExport/Products/Product/ProductVariations/ProductVariation

It will now find the other things that are in the same level as ProductVariation like:
ProductId (product_sku) and SupplierProductNumber (product_mpn)

It will however not find things that have subsets like Barcode (product_gtin) or Attributes (custom_title and custom_value).

3. Path: ProductExport/Products/Product/ProductVariations/ProductVariation/Attributes/Attribute
   

Setting It this way will fill in custom_title (Attributes/Attribute/Name) with the first (Color) of two attributes. But if I make a combine rule to get the 2nd (Size), it will not find it.

<Attributes>
<Attribute>
<Name>Color</Name>
<Code/>
<Substitute/>
<Value>pet bru bru</Value>
</Attribute>
<Attribute>
<Name>Size</Name>
<SortOrder>0</SortOrder>
<Substitute/>
<Value>L</Value>
</Attribute>

4 Path: ProductExport/Products/Product/ProductVariations/ProductVariation/Barcodes

It will now find the actual value of Barcodes/Barcode/BarcodeOrder "1" but other things stop working.

So unfortunately this is not working yet. What I think is good, is that the script knows that there are multiple values for, for example, images. This way you could probably say, take the first and the 3rd image. Or, only use the first of 2 Barcodes.

The potential good thing is that you can set it to a deeper path but still get values from paths that are higher up. So when you would set it to Productvariation, you still can get the category/ProductGroup and the ProductNumber of the Parent product making it possible to import custom fields, parent and child products in one import.

Cheers

Hi Roland,

"Are you only able to import 3 categories with the patch I sent?"

Yes, in the live site and in the test site.

Below is talking about situation without the latest patch:

The problem with products is that you have a parent product and child products.
The first ProductVariation of every Product is basically the Parent. All other ProductVariations are childs.

When you set the node to Product
It will only see the first ProductVariation for every Product. You can combine fields but with ProductId (=product_sku) that's not an option. Unless you can combine those and use them in custom_param for customfields (like I've mentioned before).

When you set the node to ProductVariation
It will not be able to tell what the Parent product is.
Another problem, it will lack the information that is one level higher up like category (Groups/ProductGroup), manufacturer (Brand) etc.

Cheers,
Roderic

Hi,

I was just working on creating a post, making screenshots and getting confused on all the different things I've tried. So hopefully it won't be neccesary anymore as that would save a lot of headaches! :D

Waiting impatiently

Hi Roland,

Any chance of the patch coming today?

Cheers!

Hi Roland,

Since I've already manually added all the groups + made a rule to replace the GroupGuid with Virtuemart ID, I don't need the group part for now (untill they add groups in the future,). So yeah, products is definitely the thing I'm waiting for. :)

edit: Oh and also need the groups import for the groups description and meta information but that's not a prio at the moment, products is.

Hi Roland,

Thanks for the patch. I've tried it on the live site and test site with the suggested path and am getting no output at all.

Template fields:

Preview:

After import

Analyzer

Not sure what I am doing wrong?

Cheers

Good evening,

I have set the import to a FTP folder where Vendit automatically exports new XML files every day. But they should not be different. I downloaded the latest one and attached it here.

Cheers

• Products(2).xml

Good evening once more,

So I've added almost all required fields to the template in the test website. It seems I can get values from almost all required fields except two.These have subpaths.

Some values that are empty below, are empty because they are empty in the XML for that product. With only one product being imported, it's difficult to test but my feeling is it will work. Just those 2 with subpaths won't.

I hope this is helpfull information in any way.

Have a good weekend.

Hello Roland,

So this looks very promising. I imported the Products.xml on the test environment and:

• All products have been imported with all the information available.
• Childs have been imported as Childs of Parents.
• I can update the stock with Stock.xml

You can run the CHILD and PARENT templates on the test site if you like.

But.. I don't see how I can import stockable fields. I made a copy of the template and changed the path, this way it will only import Parent products. But I think the way you have set up your breakdesigns plug-in, it will only work when you import Parents and Childs within the same import (as in the example/tutorial with CSV file) So this is how it looks in the preview:

Now with the Stockable plugin you can add existing child products as stockable manually:

If I would first import the Parent product, then import the childs with another template and your breakdesign plugin would create the stockable fields based on the amount of childs being imported. Wouldn't that make the most sense?

I ran into the same problem with an Excel file from a bike shoe manufacturer last week, see attachment. Then this part in the tutorial for custom_value:

This must always be stockablecustomfields for the parent product, as many times as there are child products. Each value separated with the ~-symbol.

Now there are shoe models with like 60 colors and size combinations. However I see no way of gathering the amount child products and thus values of stockablefields. I managed to import the child / parent products + customfieldsforall (color+size) but then again ran into the problem of having to assign the stockable childs manually. Unless I'm overlooking something in both situations.

So yeah, conclusion, pretty close, but not there yet. Now the question is, can the desired result be achieved with what is currently available?

Cheers!

• 2021 EAN UPC info sheet revised 1.12.2021.xlsx

Hi Tharuna,

Yes I was aware what it does. I've got it all setup properly in the Parent products import template:

This will indeed import the Parent products and add X amount of rows in #__virtuemart_product_customfields table.

And this is how my Child import looks:

The child products will be added to the proper parent products but the rows in #__virtuemart_product_customfields table stay untouched.

That's why I figured this will only work when you do this import of Parent + Childs in one and the same import/template. Or am I wrong?

Cheers

Hi Tharuna,

Your example is the example for normal customfields, not customfieldsforall. There is an example in the same tutorial:

It is also possible to import customfieldsforall with stockablecustomfields in one import file with proper use of custom fields. An example would be like below

"product_sku";"product_parent_sku";"custom_title";"custom_value";"custom_multiple";"custom_param";"product_name";"product_in_stock";"product_price"
"10001";"";"Type~Material~Size and Color~Size and Color~Size and Color";"customfieldsforall~customfieldsforall~stockablecustomfields~stockablecustomfields~stockablecustomfields";"~~Y~Y~Y";"Semi soft~Plastic";"Product with filter";"0";"100"
"10001brbl";"10001";"Type~Material~Size~Color";"customfieldsforall~customfieldsforall~M~Green";"";"Soft~Foam";"Child Medium Green";
"10001brge";"10001";"Type~Material~Size~Color";"customfieldsforall~customfieldsforall~L~Red";"";"Hard~Metal";"Child Medium Red";
"10001bror";"10001";"Size~Color";"XL~Orange";"";"";"Child Extra Large Orange";

So my templates are pretty much setup as in this example, right?

Hi Tharuna, thanks, that got me a big step further. Now the next problem:

It randomly add's ID's as you can see below, 2 different products.

Now I've made an export and then I noticed the following:

As you can see, it will fill in child_product_id="1301" with parent product_sku 30000408. This is correct.
Now the next ID, 1302 is connected to 30000193, but this should also be connected to 30000408.

So somehow it's adding the child ID's vertically instead of horizontally. Is this an actual bug or something I've done wrong?

Almost there! :D

Good evening,

I've messed around with it for several hours and unfortunately getting the same result. Maybe one of you can have a look at my test environment and see for yourself?

Maybe it's a bug, maybe it's because of incompatibility with the patches, or maybe I'm overseeing something?

This seems to be the last hurdle..

Cheers!

Hello Tharuna,

Yes you can delete the products, but I just did that, ran the import and the patch seem to have fixed it :D

As you can see below, a logical order of sizes instead of numbers and letters combined and only one Color.

So, now the question is, how to get these patches? Aka, how to get this on the Livesite?

Also, When you export a template, some rules are copied over but sometimes the values of these rules are not exported or imported. For example, I exported the product import from the live site where I've made a conditional rule for every GroupID = VM Category. The rule is copied over to the test site but without the values. This rule has a lot of values (60+) so maybe there is a limit?

Cheers!

Good morning!

So I worked on a site with CSV import with Cf4all and stockablecustomfields, got that all working and learned a few things from that. Then went back to this XML import story. A few things:

1. Exporting any template with rules gives me an error:

2. Is there a way to get only the first productId of productvariations? So basically the first child?
   2.a. If yes, I can set this as parent_sku and make the parent orderable

3. I have made a template for stock updates (attached), when I run this template, it will keep on running it untill I cancel it. It seems to update the same products over and over. There are only 326 products.The cron seems to run fine.

4. All the Parent products are imported with stockable, except two products. I can't see any logical reason. I use a random field that every productvariation has to count the amount of child products. Based on this rule, I made a rule that makes that amount of stockablecustomfields+custom titles+custom multiples. This works for every other parent product. In the analyzer I can see this field also shows up for the 2 parent products with issues. So am a bit lost here. Can't find a clue in the logs as well. You can test this in the test website. The products are the 2 Asics shoes.

5. Not so important atm but any update on Groups/Categories?

Kind regards,

Roderic

• csvi_templates_www.woest-sport.nl_20210430(1).xml
• Stock.xml

Hello again,

I was on a holiday the past week so apologies for late reply. During my holiday I did find some time here and there to look at some of the issues and I managed to fix a few.

We have never seen that error before nor any user reported such an error. I am thinking it is something very specific to your site. In the Destination options try Save on server instead of Download. You can then take the XML file from server location. For download you can check with another browser or with incognito window. See if that works.

Going to Maintenance -> Backup templates and choosing save on server instead of download works, I got the error when exporting from Templates page (both test as well as live site) where you can not choose for download or save on server.

2. Is there a way to get only the first productId of productvariations? So basically the first child?
   2.a. If yes, I can set this as parent_sku and make the parent orderable

You can use Skip template field for fields you need to ignore on import. So use parent_sku from first product id of productvariations and for the rest you can set it to skip template field. Did you try that?

But how does it know what is the first product of a container?

4. All the Parent products are imported with stockable, except two products. I can't see any logical reason. I use a random field that every productvariation has to count the amount of child products. Based on this rule, I made a rule that makes that amount of stockablecustomfields+custom titles+custom multiples. This works for every other parent product. In the analyzer I can see this field also shows up for the 2 parent products with issues. So am a bit lost here. Can't find a clue in the logs as well. You can test this in the test website. The products are the 2 Asics shoes.

I don't understand how you can see stockablecustomfields+custom titles+custom multiples values of products in analyzer and that too with rules. Can you post a screenshot of it? Meanwhile i am checking the import of parents on your site for these two products.

So I fixed the problem and reworked the way I got the amount of stockablecustomfields+custom titles+custom multiples.

Every child has a field called:

<IsMandatoryExtra>False</IsMandatoryExtra>

I combine all these with a combine rule. Then I replace every False with Maat en Kleur for custom_titles. Then I copy this field to custom_multiple with a copy rule and replace every Maat en Kleur to Y with Find and replace rule. Afterwards I copy this field to custom_values with another copy rule and replace every Y with stockablecustomfields with another find and replace rule.

So i ran import in your site to check these two products. The corresponding product_skus are 151927-296 and 166419-814. I ran import of parent import template on your site to check the latest debug log. I see that for product 166419-814 the values are imported. For product 151927-296, there is an error in debug log "The number of custom titles is not equal to the number of custom values", this means the count of values in custom_title and custom_value is not same for this product and so nothing is processed. You need to check why the count is going wrong for this product.

So that's how I get those values and that works smoothly now.

Just an update on the endless loop, this happens because RO CSVI can't keep track of the last record. What I saw in your template is that you have set the number of items per import to 250 while the import file is 280. So if you change the lines per import to 300, your import will finish. The 300 number is fine for just the stock as it is a fast import.

Yes, this change helped.

At the moment I have redone the XML reading one more time and am happy with how it is performing and everything is working as expected. The XML reading is now able to process your Products file with the container nodes and product nodes. So the products have the data from the container node. The plain XMLs are also processed by this code ensuring older imports keep working.

I've got everything working when it comes to Products.xml. Except for having the first child product as parent product. I thought this was neccesary to have only the parent showing up instead of every child. I deleted all products once more, imported the parents with and the childs without category. This way I only get to see all the parents in category view. This is good enough although it would also be nice to have the parent to be orderable. That way when you click on a parent, you can already have a stockable product selected.

So the question is now, will your update break what's working now? :D

Thanks!

Hi Tharuna,

Hello Roderic,

Going to Maintenance -> Backup templates and choosing save on server instead of download works, I got the error when exporting from Templates page (both test as well as live site) where you can not choose for download or save on server.

For some reason download from your site does not work. From Templates page, there is only download option and so it breaks on your site.

This stopped working after the last 2 patches.

Wow, power of RO CSVI rules:D

Indeed, now if I could figure out how the expression properly works in the find and replace rule, I could potentially optimize it even further. There is no documentation for this right?

If possible send me an example from Products.xml file, i will see if i can explain it better with the example.

Attached

So the question is now, will your update break what's working now? :D

The one thing that will break is that the template field with the xml_record_start is now renamed to xml_record_container so you need to enter the start record again once we release the new version or send you a patch file. Nothing else should break as you are using full paths already.[/quote]

That sounds doable :D

Cheers,
Roderic

• Products(5).xml

Hi,

After a few attempts in the past few weeks of writing a reply, here it finally is:

My client has been adding products every day and I've been running the import on a daily basis. Ran into some problems but managed to fix most of them myself. For example, you need to add has_medias = 1 to child products if you want them to have images without (re)processing the images but using Update media link only. Maybe this information could be added to the Help text in the edit template screen?

There are multiple options for the image name, what I'm missing is the option to fill the media fields with with a different value. In my case the image name based on the product ID is perfect, but this same ID is useless for the media fields. I would like to use the product name for that.

Then another thing with media, I'm importing the images both for parent as child.
XML:

<Images>
<Image ImageOrder="0">A7F54BC3-E4F0-414D-A8FC-F47624B20FB4.jpg</Image>
<Image ImageOrder="1">9E996D7C-52BA-415E-80F5-5E662DC5FC07.jpg</Image>
</Images>

Template field:

ProductExport/Products/Product/ProductVariations/ProductVariation/Images/Image

The amount of images per product varies from 1 to 10 and everything in between. So with combine fields I'm able to import them all. For the child products this works perfect. For the parent products it will see all the images from all the ProductVariations. This means it will import the same images multiple times:

In the end it will not show the images double, but it might process the same image over and over and thus adding unnecessary import time. This is however not a big problem but the actual problem is, when the amount of images of a child product is uneven, it will mess up the order and thus pick the wrong image as the first image. This sometimes leads to the wrong image showing up in browse-view. Any workaround for this? Either CSVI needs to be smart enough to see the file_url is exactly the same and thus ignore the duplicate value, or somehow need to be able to influence the image order.

When I open an imported product, it will default the layout to default even though I've set this to be a custom template (woest) in the VM Config. To solve this I've added layout as a field in the import template and set the default value to woest. That fixed it. But why does it change the value? Is this a CSVI issue or Virtuemart?

When using the Customfilters Pro module from Breakdesigns, click on a filter that's only part of either a child or parent product, it will not show the other filters.
For example, the child products have the colors and size and the parent product has the other custom fields. When I select a color filter, it will hide the customfields from the parent product. If however I click on a filter from the parent product, it will hide the color and size filters. Meaning you will have to import all the customfields for both parent & child (and make use of custom_override otherwise it will show the custom fields twice in product-detail view). This wasn't clear to me when following the tutorial. Maybe also something to add to the breakdesigns documentations?

Sticking with Breakdesigns, I discovered by mistake that one doesn't need to use hex colors to be able to show colors in the color filter. You can also use HTML Color Names. This way you don't need a rule to replace the color names to hex with CSVI Rules find & replace plug-in like I was doing. Makes things quite a bit easier. CF4A then needs to be set like:

We do have a documentation for RO CSVI Multi replace rule.

Yes I've checked this documentation before but it doesn't give me an example of how to use it with RegEx.

For example, in the Find and replace rule below, I would like to make use of an Regex rule. So I would only have to use customfieldsforall once with a clever rule. If this is possible, could you give me an example?

Is the error for specific templates or all templates? The patches sent to you are not related to backup template feature so i really doubt if it is because of the patch file.

It seems it was only a problem with a few templates that were using a specific rule. Also, whitespaces in rules are not exported (or imported, not sure). Like in the below example:

That's it for now :-D

Cheers!

Good afternoon,

Thats correct with Update media link only option we are creating a link between product and image which also means product has media. Will correct the code to automatically set has_media field to 1. That will be a better solution than adding it to help text.

That indeed sounds like the best and most logical solution.

When you set Fill media fields to yes, image name is used for media fields, in your case it is product ID. If you want product name to be used then set Fill media fields to No, create a Field copy rule to copy product name to media fields.

In this case I also think it's a better solution (also more user friendly) to have the option in the template image options to choose a different option for image name and media fields. Like this:

The problem in my case is as well that you need to set the file_title / file_description for every media file. But when you use combine to get multiple images, how to get multiple values for file_title / file_description as well?

So because of that I think the fore-mentioned solution is the solution. Or else, see below.

Regex are used to search and replace string with a specific pattern. In your case you are trying to replace different values with one value. I don't think regex will help here.

I see. You've put me in the right direction there on using regex and I've been trying some things today.

This will replace every file_url I've combined into "image". Now if I could replace it with the value of product_name instead, like you can with conditional field plugin, I would achieve the desired result as well. This could also be helpful in many other situations.

When inserting an image, RO CSVI checks if the image already exists. If the image does not exist then inserts it else existing image is linked with product. For setting a order of image you can use file_ordering field with multiple values set as 1|2|3|4|5...

This gives the same problem as above. But even when I set this as default value, it unfortunately won't help. I've tried many things.

Cheers,
Roderic

Hi Tharuna,

Attached patch file will update has_medias field when Update media links is set to Yes. Load the patch file, try the import and let me know.

Thanks, will try later.

May i ask why not set Image name based on to Product name field instead of Product ID? This way with Fill media fields set to Yes, image name will be used for file_title / file_description which will be product name and you will achieve it in a simpler way than with regex rule.

Because when I try to import with the setting set to Product name, and you want it to have an effect, you need to set Create image name to yes, it will then not be able to find the images. I know realize the Fill media fields doesn't obey the Image name based on when Create image name is set to no.

So the actual problem might be that with Create image name set to yes, it tries to rewrite the filename although it maybe should only set the file_title?

Warning: getimagesize(/pathtomywebsite/images/virtuemart/product/Specialized_Recon_2.0_Mountainbike_Schoenen_1.jpg): failed to open stream: No such file or directory in /pathtomywebsite/plugins/csviaddon/virtuemart/com_virtuemart/model/import/product.php on line 1777

You can login to the test site and change the settings on the parent template and run a few tests if you like.

The problem here is that with Find and Replace you can replace with a given text but not with a value of field. With conditional plugin you can check for a value and replace with value from product_name field but in your case it is multiple values combined with |. Check if Image name based on product name solution will help in solving this issue.

Yes, that's what I tried to describe, you've used a few less words ;) So I'm asking to have one of the plugins changed/updated to make them (and thus also CSVI in general) even more powerful. Not just for this situation.

You mean setting custom_ordering field does not work for you? Can you explain this in detail?

You mean file_ordering. So, here is an example product (trimmed):

<Product>
<PageTitle>Brooks Glycerin 19 dames</PageTitle>
<ProductVariations>
<ProductVariation>
<ProductId>166415-782</ProductId>
<Size>7</Size>
<Color>Black/Ombre/Primrose</Color>
<Images>
<Image ImageOrder="0">CDBD539B-A69C-4300-A4A7-F549A66B7EB3.jpg</Image>
<Image ImageOrder="1">69504C8E-B898-4020-B01B-4D924258A805.jpg</Image>
<Image ImageOrder="2">406681BB-A258-49B3-964F-31992ABE5381.jpg</Image>
<Image ImageOrder="3">EB624DD2-179A-4900-B23F-85C5622B3DC2.jpg</Image>
<Image ImageOrder="4">A75E6074-A5A6-49CB-9A9F-9EBDA2F5EABD.jpg</Image>
<Image ImageOrder="5">C6966F9E-4581-4EC9-92C8-014C328D3ABF.jpg</Image>
</Images>
</ProductVariation>
<ProductVariation>
<ProductId>166415-783</ProductId>
<Size>8</Size>
<Color>Black/Ombre/Primrose</Color>
<Images>
<Image ImageOrder="0">CDBD539B-A69C-4300-A4A7-F549A66B7EB3.jpg</Image>
<Image ImageOrder="1">69504C8E-B898-4020-B01B-4D924258A805.jpg</Image>
<Image ImageOrder="2">406681BB-A258-49B3-964F-31992ABE5381.jpg</Image>
<Image ImageOrder="3">EB624DD2-179A-4900-B23F-85C5622B3DC2.jpg</Image>
<Image ImageOrder="4">A75E6074-A5A6-49CB-9A9F-9EBDA2F5EABD.jpg</Image>
<Image ImageOrder="5">C6966F9E-4581-4EC9-92C8-014C328D3ABF.jpg</Image>
</Images>
</ProductVariation>
<ProductVariation>
<ProductId>166415-784</ProductId>
<Size>9</Size>
<Color>Black/Ombre/Primrose</Color>
<Images>
<Image ImageOrder="0">CDBD539B-A69C-4300-A4A7-F549A66B7EB3.jpg</Image>
<Image ImageOrder="1">69504C8E-B898-4020-B01B-4D924258A805.jpg</Image>
<Image ImageOrder="2">406681BB-A258-49B3-964F-31992ABE5381.jpg</Image>
<Image ImageOrder="3">EB624DD2-179A-4900-B23F-85C5622B3DC2.jpg</Image>
<Image ImageOrder="4">A75E6074-A5A6-49CB-9A9F-9EBDA2F5EABD.jpg</Image>
<Image ImageOrder="5">C6966F9E-4581-4EC9-92C8-014C328D3ABF.jpg</Image>
</Images>
</ProductVariation>
</product>

As you can see, every product variation has the same set of images. In this case 6 images but it can be any number from 1 to max 10. So I've set the parent template like:

Resulting in this import preview:

CDBD539B-A69C-4300-A4A7-F549A66B7EB3.jpg|69504C8E-B898-4020-B01B-4D924258A805.jpg|406681BB-A258-49B3-964F-31992ABE5381.jpg|EB624DD2-179A-4900-B23F-85C5622B3DC2.jpg|A75E6074-A5A6-49CB-9A9F-9EBDA2F5EABD.jpg|C6966F9E-4581-4EC9-92C8-014C328D3ABF.jpg|CDBD539B-A69C-4300-A4A7-F549A66B7EB3.jpg|69504C8E-B898-4020-B01B-4D924258A805.jpg|406681BB-A258-49B3-964F-31992ABE5381.jpg|EB624DD2-179A-4900-B23F-85C5622B3DC2.jpg

As you can see, the 6 images + 4 duplicates. The import results in:

Now when I change the the file_ordering to be file_title, I can show the problem:

The first set of image values are ignored (or overwritten) and it starts with the 2nd set instead. If the import could be changed that it actually ignores the duplicate values instead, that would fix this issue.

Cheers,

Roderic

Hi again,

Thats correct, you need to set Create image name to yes and Image name based on Product Name. This will then change the image name and based on this new image name(product name) file_title/file_description will be created.

Yes I figured it should work like that, but then I get the error message I posted in my previous post:

Warning: getimagesize(/pathtomywebsite/images/virtuemart/product/Specialized_Recon_2.0_Mountainbike_Schoenen_1.jpg): failed to open stream: No such file or directory in /pathtomywebsite/plugins/csviaddon/virtuemart/com_virtuemart/model/import/product.php on line 1777

In your parent product import template file_ordering field is empty

Currently yes, but I had all kind of values there.

i see you have ImageOrder="0" field with ordering value. You can use this field with combine template fields and combine values like you did for images

I actually tried this yesterday but it didn't work.

Ok after some more testing, It does work, but it doesn't accept 0's. So I made a rule to replace the 0 values with a 1. Then another rule to turn the existing 1 to a number higher than 10 so it won't override the real 1. Now the 0 in the XML file is the real number 1 (VM media file ordering starts with 1).

Cheers!

Good morning,

The error says that image in the location /pathtomywebsite/images/virtuemart/product/Specialized_Recon_2.0_Mountainbike_Schoenen_1.jpg does not exists. Can you check if you have that image in the path?

Yes I can read ;) and no the image is not in that path, that is what I was trying to explain. The filenames are like:

<Image ImageOrder="0">CDBD539B-A69C-4300-A4A7-F549A66B7EB3.jpg</Image>

So when I have Create image name set to No, the images are imported. As soon as I turn it On, it will look for non-existing pictures. As if it skips the actual renaming of files, then trying to import them. It also happens when I choose product_id:

Warning: getimagesize(images/virtuemart/product/7952_1.jpg): failed to open stream: No such file or directory in /plugins/csviaddon/virtuemart/com_virtuemart/model/import/product.php on line 1777

So expectation:
RO CSVI renames the files to the chosen setting, then imports them to the database with that name.

What seems to happen?
RO CSVI skips renaming the actual file and then tries to import the files that then logically do not exist.

Glad to know you found a workaround for that. So the issue with file_ordering field is also solved?

Yes, today I've implemented it also on the live site and improved it a bit. I made a conditional rule that makes every number +1. This way the problem with 0's is solved and the order stays the same.

Cheers!

Hi Tharuna,

In your case it was set to JPG and so rename of image did not happen. That lead to Image does not exist error.

So if it would've been a PNG/GIF, it would have converted it and changed the name?

So i changed that setting and tried the import. Now i am seeing 500 Internal server error. This error does not give any details on why it is happening. Can you check your server error logs and get the details of this error? You can post it here so we can check.

mod_fcgid: stderr: PHP Warning: getimagesize(--/images/virtuemart/product/suplest_edge+_xc/gravel_performance_xc/gravel_mountainbike_schoenen_4.jpg): failed to open stream: No such file or directory in ---/plugins/csviaddon/virtuemart/com_virtuemart/model/import/product.php on line 1780, referer: https://csvi.woest-sport.nl/administrator/index.php?option=com_csvi&view=import

Ok so it seems it does work but it runs into an error because of the + sign. I guess I will need a patch that filters out the plus?

Roderic

Hi Tharuna,

I see the + sign in the path of the image. May i ask where is the suplest_edge+_xc folder coming from?

It's simply the (rather long and stupid) Product name.

And maybe the forward slash in the product name is the actual problem then.

	mod_fcgid: stderr: PHP Warning: getimagesize(/images/virtuemart/product/Shimano_Ultegra/XT_HG93_9-Speed_Ketting.jpg): failed to open stream: No such file

Link to product

Thanks

Hello Roderic,

And maybe the forward slash in the product name is the actual problem then.

Yes, the forward slash is the problem not the +. With the slash the name is considered as folder name and so it says image does not exists. Try to replace / with any character withRO CSVI Multi replace rule and see if it solves the problem.

I did after my post, and the import runs without error.

https://csvi.woest-sport.nl/en/bikes/remmen/suplest-edge-sport-crosscountry-mountainbike-schoenen-detail

But as you can see, images are being duplicated. Before it would see that the image is the same, but now with renaming, it thinks every image is unique...

So, any idea how to fix this?

Cheers

Hi again,

Another problem that I almost forgot about:

When my client changes the color, the new color gets imported but the old value stays. (Delete custom fields relations set to yes)

When I open the product in the backend, it doesn't list the old color value:

So then when I hit save, the old value is gone. How to trigger this automatically?

Roderic

Hello Tharuna,

Can you send me the debug log of this import to check?

Attached

With delete custom field relation set to yes, we were not deleting values from custom fields for all tables and so old values were left. Load the attached patch file and run the import again. This should remove the old values and insert the new values for products.

I installed the patch on the test site and now get this error:

2021-06-04 	13:08:17 	4 	[QUERY] 	SELECT `customsforall_value_id` FROM `xxx_virtuemart_product_custom_plg_customsforall` WHERE `virtuemart_product_id` = 7954 AND `customfield_id` = 494634
2021-06-04 	13:08:17 	0 	[DEBUG] 	You have an error in your SQL syntax; check the manual that corresponds to your DB server version for the right syntax to use near ')' at line 3
2021-06-04 	13:08:17 	0 	[QUERY] 	DELETE FROM `xxx_virtuemart_custom_plg_customsforall_values` WHERE `customsforall_value_id` IN ( )

Cheers

• com_csvi.log.470.rar

I have installed updated patch on your test site. Please check import and let me know.

That worked, thanks!

There is no direct way to do this but you can write some custom code and create a RO CSVI rule plugin to read values of file_url field, remove duplicates and update the value to file_url field.

That sounds complicated but I can give it a go.

Have a good weekend!

Hello,

Been a while! In the meantime this import has been running every day and mostly without problems. The import of Groups.xml (Categories) has never worked however and I was wondering if there has been any improvements on that part.

I was hoping that all the patches from this topic would've made it into the new version. But it seems they haven't. Now it is a bit hard to know which functionality did and what didn't. Do I need to repatch after updating?

Also, on my test website I've got Joomla 4 running and installed beta versions of Phoca and VM. I get 500 internal errors when trying to run a template with VM or Phoca fields. Is this expected?

Cheers,
Roderic

Hi Roland,

Ok seems I've fixed a few things. Now I'm running into the following problems:

When I set the child products import to what it should be, it will not see the Product SKU from the ProductVariants:

Same story of course for all the other fields in ProductVariants/ProductVariant

When I look in the Analyzer, this field does show up but the first ones are empty:

Then when I scroll further, the fields have data

• The import seems to stop at possible legit errors that it used to ignore. For example, there seemed to be one or 2 duplicate SKU's that I've now deleted.

After several import attempts and fixes,The Parent product import on the front-end successfully finished. I just ran the cronjob as well and that also managed to finish because I fixed those problems. Before it would just throw up a fatal error and say it ran out of memory and not show the actual error.

Cheers

Hello,

Attached is the latest Products.xml.

Also attached are the 3 templates I'm using (Parent (Works) Child (broken) Stock (Works) and a test template with only parent and product SKU that also doesn't show the product sku's.

Cheers

• Products.zip
• csvi_templates_www.woest-sport.nl_20211111.xml

Hi Tharuna,

When I try to import, I get this error:

Both on live site (Joomla 3 + CSVI 8 rc1) as well on test site (J4 + 8rc1).

I've looked into the xml file, I see you have changed the container to empty and the record name to Product. I'm trying to reproduce that now.

Cheers

I guess you are trying to restore the template XML zip file. Can you unzip the attached file and restore the template XML file?

Oops, yes. Didn't realize this is not possible

Not only Record container and Record name but the XML path in template fields and XML path in rules are also changed.

I see. The thing is, this way it will only see the 643 Parent products:

But with all the ProductVariations (Childs), there are 3702 products:

Now the problem is when I set the Record Name to ProductVariation. The container will change to that what you see in the screenshot. If I try to override this in the settings by setting the Container to ProductExport/Products/Product, it will ignore/override this setting.

This way the nodes that are above /ProductVariatons/ProductVariation cannot be read out.

Cheers

Ok, i did not see this. I used a part of your Products.xml file as i had difficulty in importing the whole file.

You probably didn't upload the file to FTP?

Yes, when you set record name to ProductVariation, the values are limited inside this node and any values outside the node cannot be read.

Yes, but isn't that where the setting for Record Container comes in so it can still read values outside the Record name. This is how it worked in CSVI 7.20 but somehow this is broken in 8. That's why I thought I needed some of the patches again but then realized this is (now) basic functionality that should work without a patch.

I am realising that when loading the file with Analyser, it reads the node name but not the value. That should not happen. It should read both the node name and the node values. I am looking into it and will post you an update on my findings.

Probably related to what I said above.

Cheers!

Hello,

So I've updated again to the latest version, applied the patch, ran into a few problems but I was able to fix them. Now I've ran a parent + child import and the products seem to have imported properly. The Parent import takes way longer than before however. Almost ten minutes.

There is a record with "Last modified". Is there a way so that it would skip products with last modified date before today or something similar?

I've applied custom CSS styling to some CF4A fields. Those fields got a new ID and now that styling is gone. I am not sure if this is a one off because of a few changes or that this will happen every time now because it deletes customfields instead of relations only?

Cheers

If you are looking at skipping record with one single date then you can set that in values to match field in RO CSVI Skip rule but having a check if the date is less than today is not possible with skip rule plugin. Even with single date, you need to change date value everytime you run the import.

Yeah it only makes sense when it's dynamic. So no alternative way?

With VirtueMart product import we delete the relation and not the custom field itself. Are you sure this is happening with RO CSVI imports?

Yes you would say so and it also worked like that in the half a year. It seems though that since the update, that has changed.

Cheers

Hi Tharuna,

Both logs of the automatic import from this morning attached. Parent and child. It happens to all custom fields basically.

I removed all the products from the Products.xml file but one so I could test the import a few times. This product has one customfield called Materiaal:. with value Yellow.

I ran the Child product template 3 times and checked the ID of value Yellow in the database.
Table: xxx_virtuemart_custom_plg_customsforall_values:

The ID changed after every import:
7089
7092
7095

Table: xxx_virtuemart_product_custom_plg_customsforall:

When I turn off "Delete customfields relations" in Child product, run the import, the ID stays the same (7095).

So, that's not correct I would say?

Cheers

• logs-woest.rar

Hello,

Another problem came up on Friday night. I made some changes to the import, ran the import a few times and everything seemed to work fine. Then the next morning I found out that the automatic import during the night only imported 400 child products. Parent import went fine. No errors in the log files.

I investigated the problem and after a long search (no idea where to start looking) and a lot of testing (5 hours) I found out that the import stopped on deleted products that do not have ProductVariations. I removed the first product without ProductVariations in the XML file and ran the import. This time it imported a few hundred more products. Then I removed all products without variations and now the whole XML file got imported.

I also checked a few older XML files to see if those products were in there as well, they were. So before these type of products did not stop the import.

I am also not sure why it did work on Friday and stopped working overnight. At first I thought something might have changed in the new XML file that gets send to the server overnight. But as I said before, I've then tested with the XML file that I've attached before in this topic and that didn't make a difference.

Cheers.

Hello Tharuna,

The custom fields are stored in #___virtuemart_customs table and the relations of custom fields for all are stored in #__virtuemart_product_custom_plg_customsforall and #___virtuemart_custom_plg_customsforall_values tables. That is the reason when you set Delete custom relations to Yes, the values in these two tables are deleted and inserted as new rows which then gets new IDs.

Maybe they weren't properly deleted before and now they are. So that means the ID's will always change with this setting on and custom styling for a customfield is then impossible. I'll have to find a workaround or talk with Breakdesigns, If they can somehow implement a css class system that's based on something else then ID's.

What do you mean by deleted products? The Products.xml file you sent has any such products? Can you give an example from the file you sent us?

So products that are deleted from Vendit, are exported with

<IsDeleted>True</IsDeleted>

For example:

<Product>
			<EcommerceProductGuid>2BA59D70-F225-451A-9608-DC86CB4C4A16</EcommerceProductGuid>
			<ProductNumber>4250773365691</ProductNumber>
			<Description>Aspero 5 GRX RX815 DI2</Description>
			<Type/>
			<Brand>Cervélo</Brand>
			<SmallInfo/>
			<BigInfo/>
			<Occasion>False</Occasion>
			<CompositeProduct>False</CompositeProduct>
			<FrontPage>False</FrontPage>
			<PageTitle>Cervélo Aspero 5 GRX RX815 DI2</PageTitle>
			<MetaKeywords>4250773365691, Aspero, 5, GRX, RX815, DI2, Aspero 5 GRX RX815 DI2</MetaKeywords>
			<MetaDescription/>
			<Visible>True</Visible>
			<AvailableFrom>2012-01-01T00:00:00</AvailableFrom>
			<AvailableUntil>9999-12-31T23:59:59</AvailableUntil>
			<OrderableFrom>2012-01-01T00:00:00</OrderableFrom>
			<OrderableUntil>9999-12-31T23:59:59</OrderableUntil>
			<OrderabilityRemark/>
			<StockProduct>True</StockProduct>
			<ShippingCost>0.0000</ShippingCost>
			<IsDeleted>True</IsDeleted>
			<LastModified>2021-11-09T14:07:23.023</LastModified>
		</Product>

As you can see, this product does not have ProductVariations anymore. There are also Products with ProductVariations with IsDeleted = true but these are not the problem. So the problem is the lack of missing <ProductVariations><ProductVariation> and that makes the import stop. Maybe because it thinks it reached the end of the file?

Cheers

Hello Tharuna,

Before I did import those products, if IsDeleted = true, I have made a rule to make True to be "1" and then those products would get deleted when running the import. Worked fine.

Now when this problem arised on Saturday, one of the things I tried was to use the Skip rule to skip records based on IsDeleted = true as that is the common node. This unfortunately didn't help.

Next problem, I'm trying to improve the import of prices. Before I would import the discounted prices during Parent and Child import. Now I'm importing the normal price only and then importing the Discounted price ActionPrices/ActionPrice afterwards in a separate import so I can import the publish_up and publish_down properly. But eventhough I first import the normal prices and afterwards the actionprices, the order is reversed in the administration and this makes that the discounted price doesn't show up. I can also not swap them around manually.

Good morning,

Here a screenshot of the rule:

I tested it again just now and it worked this time. I changed the path to Products/Product/IsDeleted instead of Products/Product/ProductVariations/ProductVariation/IsDeleted

But this is not a great solution since the deleted ProductVariations (Childs) will now not be deleted but skipped. Or are Child products also deleted when the Parent get's deleted while running the Parent import? In that case all is fine.

I duplicated the website to test, reinstalled 7.20 and ran the Child template. It didn't halt on the products without ProductVariations like it does in 8.0 rc1.

The custom fields are stored in #___virtuemart_customs table and the relations of custom fields for all are stored in #__virtuemart_product_custom_plg_customsforall and #___virtuemart_custom_plg_customsforall_values tables. That is the reason when you set Delete custom relations to Yes, the values in these two tables are deleted and inserted as new rows which then gets new IDs.

Also in 7.20 it doesn't delete values from #__virtuemart_custom_plg_customsforall_values like it does in CSVI 8 with Delete custom fields relations set to Yes.
It also shouldn't delete those values because then when you reorder those values in a logical order in the VM Admin, you'll have to do it again after every import. So it should only remove the relations from #_virtuemart_product_custom_plg_customsforall and not the actual values from #__virtuemart_custom_plg_customsforall_values Can you fix this?

There is no ordering field in VirtueMart for prices so whatever order you are seeing with prices is with VirtueMart code. From what i noticed, product prices are sorted based on the highest price. Since other prices are same but discount price is higher for one row, that is shown as first price.

When I manually add a price, it will come below the original price and then the discount works (see screenshot). When I run the import with similar values, the discounted price comes on top and doesn't show up on the front-end.

Also, there is this button next to the trash icon that enables you to sort the pricing. It allows you to drag one price-box over another. Then when you save it however, the price-boxes are back to the original position but the value for overriding has swapped. So basically there is a button but what is missing, is a column "ordering" in the VM table #_virtuemart_product_prices.

Cheers!

Hello Tharuna,

No, child products will not be deleted but the parent child reference will be reset when a parent is deleted. The product_parent_id field value will be set to 0 for all the child products when a parent is deleted.

Wouldn't this be a good option to have in the template settings? "On Parent product deletion, include deletion of childs: Yes/No"

You can have another import to delete the child products. Will that not work?

Sometimes the ProductVariations are missing and only the Parent product with IsDeleted=True is left, in this case this will not work. Otherwise yes, a separate import to delete child could work.

There were improvements made in code to read values from ProductVariations/ProductVariation node and a patch file was sent to you recently. With 7.20.0 version, were you able to read node ProductVariations/ProductVariation values? For example ProductVariations/ProductVariation/ProductId field.

Yes, in 7.20.0 this worked/works well with patchespatches applied from this topic. Version 8 broke this functionality but then the patch you mentioned brought back the functionality I had in 7.20.0+patches for the most part. But it's not 100% the same. I just read through a big part of this topic and I feel that most of the changes that were made in patches attached in this topic made it into 8.0.0 but not all of it.

The issue of old values staying in database and creating conflicts was reported by you and was fixed for you. See attached screenshots and reply for it. Resetting the code will get the old issue back.

Yes you are right. I am not sure if I used that patch on the live site in the last 4-5 months. I think I did not. That's why the customfield values stayed the same.
With delete customfields relations set to yes, What it should do, is remove values from:
#__virtuemart_product_customfields
#__virtuemart_product_custom_plg_customfieldsforall

What it should not do, is remove values from:
#__virtuemart_custom_plg_customfsforall_values

Why not?

• This messes up the order of the custom values and you have to reorder
• When you delete a value, it will also be removed from another product using that value. Meaning, you always have to do a full import otherwise there might be a few products with custom values missing.

So I think resetting the code would be the best option. I will then have to test if old values remain with a product or setting delete customfields relations to yes will be enough.

I am not sure on what basis VirtueMart displays prices on frontend but with RO CSVI import, we look if there is a matching price row in database and if row exists it is updated or a new price row is inserted. The new inserted row takes the new increment ID and is added as a last row in the database table. We cannot have a row insert in-between existing rows or have any control over it.

When I was testing this, I checked the database after every import and indeed saw that the newly imported prices indeed have a incremented ID.

UPDATE

I've been testing something and came to a few conclusions that I wanted to share.

The prices are being sorted by either cost, base or finalprice. Discounted price is being ignored. Thus, when those other 3 prices are the same, there is no proper ordering.

As soon as you change the costprice, the order changes. Even when it's a 0.0001 difference. The order is then based on the setting in configuration -> prices -> Order of multiple prices

It would be good when Virtuemart actually takes the discount field into account, but it doesn't. I will report this bug (?) on the VM forum.

As a workaround I've made a Margin rule that subtracts 0.0001 from the product_price field in the Action prices template. This then helps with the sorting.

Cheers.

Good morning,

I really wonder what is different because I am not aware of giving you one patch and include another one in the release.

So, in 7.20 it was able to read all the values from ProductVariations/ProductVariation/
After the update that stopped working. Then you gave me a new patch and it worked again with a few differences. Maybe CSVI has become more strict in general? I ran into a few issues that I fixed that should've come up before as well.

Yes, it has become a huge thread, perhaps we should start a new one with what you believe is missing in the current version and we can focus on that.

I will post further questions in a new topic, except for the next one as it's related

I see what you are saying here and I believe we should extend it a bit more. Instead of deleting or not deleting, we should delete the values if there are no other products attached to it. This way you can at least do some housekeeping to keep things sort of clean.

Did you get any further with this? For the last week (or 2) I've been running the import with Delete customfield relations set to No. This at least makes sure that the custom ordering of values for every customfield stays intact. However, I'm running into the following problem because of this:

After I've imported a fresh parent product, then child, it looks like this:

That's the desired result.

Then after the child template another time and every additional time it looks like:

The functionallity in the shop still seems to be intact. However, it adds empty "Maat:" values.

Which then can be seen in the Cart and order e-mails: (looks worse after several imports)

When Delete customfield relations set to Yes, running the import deletes all the empty Maat values. But of course it also destroys the ordering like said before.

Now I'm wondering why it doesn't use the value that's already there but instead inserts a new one. This only happens with stockablecustomfields and not with customfieldsforall.

Cheers.

Hi Roland,

So I've tested this on a test site with 3 products containing the same value (One Size) for a customfield (Maat: ). I run the import with 2 of these products. After running the import, the value for the 3rd product is removed. Which it obviously shouldn't. Log is attached.

Cheers

• com_csvi.log.128.zip

Hoi Roland,

I've tested a few times and it still deletes the values. I notice that when I attach a value to a product that's not in the import, the ID of the value will stay the same after the first import. Then running the import again will remove the value.

Before first import - one product with Maat value S:

After first import:

ID of S has stayed the same but value has been removed from product. M - L - XL ID's have changed as there was no product with that value.

After 2nd import:

ID has changed because there was no more product with that value as it was disconnected after first import.

Also, besides S, M-L-XL are being imported in the first import for those products. When running the same import with the same file/products again, why delete the values?

Cheers

Hello Tharuna,

Logs attached and also xml file where I'm testing with.

Cheers

• com_csvi.log.152.zip
• com_csvi.log.153.zip
• com_csvi.log.154.zip
• Products(1)(1).xml

Hi Tharuna,

I installed the patch (twice, to be sure) but can't see a difference. Log attached.

Cheers,
Roderic

• com_csvi.log.160.zip

Hello,

With the new patch file, the values from #__virtuemart_custom_plg_customsforall_values table

You are actually right. This time my test was not as thorough as normally. I added the value S to a product that is not in the import. Then I ran the import and checked the product again afterwards and the value was removed (like before). So that's not what should happen but the value in #__virtuemart_custom_plg_customsforall_values still has the same ID. So that's good.

Cheers.

Hello,

I applied the patch, did the same thing as before and the value S still gets removed like before. I then added a different value that's not in the import file and the value remains. The values of c4all still remain the same in the database so that's still correct. Import file attached but also login details.

Cheers

• com_csvi.log.170.zip

Hello Tharuna,

That product is in the import and thus that product will keep / get it's value. I've just added value S to product 4763-0. You will see it gets removed after running the import.

Hi Tharuna,

I also did some checks and it indeed looks to be good now!
Also ran the Parent import and that also looks good.

I am now wondering why the import without "Delete custom fields" enabled doesn't look for the values already in place, if they are the same, ignore them. If it's a new value, add them. With "Delete custom fields" enabled it would then replace them. Or is there a certain reason this is not possible?

Cheers!

Hello,

So if you check the custom fields of product 119843-0, then run the child product template with Delete custom fields set to No, then check the product again after the import, you will see there is a empty value "Maat" added to the list. Every time you run the import, it will add another empty field. Maat is stockablecustomfield related c4all. I'm guessing it has to do with that.

When I run the Parent import with Delete Customfields set to No, it will not make empty values for the C4all. However it also creates empty stockablecustomfields after every import. I guess in this case it doesn't know what value it should check for. It only knows the amount of stockable/childs. Perhaps it needs to check how many stockablefields already exist and not add any more if the number in the import is the same?

Hello Tharuna,

The actual stockablecustom field is productvariant and you are importing Maat in child product import template which is a customfieldsforall and not as stockablecustom field and that is creating conflicts here.

That's correct. I do not see any way to import (all) childs during import. I would be helped when the stockablecustomfield could be added during the child import.
So, first import parent with customfields(4all) but without stockable -> then import child products template where it adds a stockablecustomfield for every child during the import. In theory this would be possible I would think since you know the Parent SKU.

Can you send me (S)FTP details of the test site you sent us so to check the values in database? You can send the details in Site Details tab under your reply.

Details added. Let me know when it works/doesn't work.

Cheers.

Hi Tharuna,

When I try this setup on the live site, it doesnt work. I think I need the patch you applied?

Cheers!

Hello Roderic,
You did not tell us what is not working. There are two things done on the test site.

Yes I could have been a bit more clear, sorry.

1. Applied a patch file to delete customfieldsforall values with Delete custom fields set to Yes. Load the attached patch file to get these changes. You need to have latest RO CSVI 8.0.0 version on your live site to apply the patch file.

That's what I meant indeed, that you changed something on the test site that I would need as a patch for the Live site.

2. The other change is with import template. You can backup the template from test site and restore it on live site to get the template changes.

It's only a few changes so it's easy to move them over manually. I also tried it this way months ago but then realized it doesn't work this way as we are dealing with customfieldsforall related to stockablefield. So we are dealing with https://rolandd.com/documentation/ro-csvi/virtuemart/stockable-variants-import (last import example). With your set-up the customfieldfieldsforall is imported fine without empty duplicates but it's not attached to the parent product as a stockable option.

Cheers

when you import the child product they should be linked with the parent product. Does that not happen?

This only happens correctly when I import the child product customfield the way I've been doing it. When I do it with your changes, it indeed doesn't happen.

So I've set it up for you that you can run the Vendit -> VirtueMart Product Import - PARENT import to import 2 few products with childs.
Then you can run Vendit -> VirtueMart - Product Import - CHILD with your changes and see it won't link them.
Then there is Vendit -> VirtueMart - Product Import - CHILD copy which is set up the way that will link them. (But create duplicates when you turn off "delete customfield relations" )

EDIT:

Extra note: The product with just one child will not have a stockable, I set it up this way a few weeks ago that when a product has only one variant, it will not make use of stockablefields. So don't be confused there :)

Cheers

Hello Tharuna,

Both are setup the same with the only difference being in the customfields.Only the empty products and products with just one productvariant will be skipped. So there's still 2 products with childs/variants imported with both templates:

Hello Tharuna,

Did you changed some settings in The Vendit -> VirtueMart - Product Import - CHILD template?

No I didn't change anything.

What is the problem in running the import with your changes in the template?

The only problem is that it creates empty custom fields when "Delete customfields relations" is set to No. I just changed that setting an ran the import twice, then the result is:

As you can see, 2 empty values Maat:. Like I mentioned in other posts.

I suppose I could just leave "Delete customfields relations" turned on for the Child template and that would keep things clean. Before this was not possible as it would delete the custom values from the custom values table and then mess up the ordering of custom values. Since that was recently with a patch, that's not an issue anymore. So maybe we should just leave this in it's current state now and close this topic. I will then open a new one if there are other issues.