Resolved RE: Unable to import stockable custom fields - VM

Hello,
I have created your issue as new topic as that other topic is very old.

Please post import debug log to check. To get the debug log see How to collect debug information? document.

HI Tharuna,

Thank You very much. I will add Debug Log in a minute. After spending all night in attempts trying to figure out what is going wrong, have done few things and did make it work at the end but there are still some bugs and issues.

I have:
- Latest version 4.x.x of Joomla
- Latest version of RO CSV and it's plugins
- PHP version 8.2

Will try to cover all my findings and with the issues I can replicate. Not all are related to my attempt to import data but overall findings which maybe belong to a Bug report.

1. Exporting and Importing templates (as a backups) does work in some sitiotions, while it's throwing errors depending on state of other tables in DB
   
2. #_virtuemart_product_custom_plg_customsforall' doesn't exist error during import if import template has "Delete custom fields relations" enabled in settings.
   
I had to reinstall RO CSV component to fix the tables and disable this feature in template settings (even if I need it). I need this feature to delete custom fields data during import (drop everything it has and insert it again). As a workaround same issue I am using "Delete related products" option which deletes related products just in order to insert fresh Custom Fields data. Please see description below.

3. Stockable Custom Field limited support for flields that contain characters in names: č ć š ž đ. I had field named "Veličina" (Size in english) which would skip the field on import.
   

In relation to "Delete related products", I am using Example VirtueMart Product Import template to import products (and update them) on daily basis.
If I don't use this feature, Root product where I have child products will get double entries for custom fields. Import does not owerwrite custom fields of existing entries but adds more on top of it.

Will add few more once I run the process again and force the errors. Log is attached below.

Hello,
Please add details in this thread as this is the new forum topic created.

Hi Tharuna,

I am sorry I don't understand what do you mean by adding details in this thread. I have added details above.

Thank you.

Hello,

tharuna means the Site Details that you can add when you add or edit a message below your message. By default the Attachments tab is opened at the bottom of each message.

Hello,

Exporting and Importing templates (as a backups) does work in some sitiotions, while it's throwing errors depending on state of other tables in DB

What is the error you see? Can you post a screenshot?

#_virtuemart_product_custom_plg_customsforall' doesn't exist error during import if import template has "Delete custom fields relations" enabled in settings.
I had to reinstall RO CSV component to fix the tables and disable this feature in template settings (even if I need it). I need this feature to delete custom fields data during import (drop everything it has and insert it again). As a workaround same issue I am using "Delete related products" option which deletes related products just in order to insert fresh Custom Fields data. Please see description below.

That table belongs to Breakdesigns customfieldsforall. That table is missing and has not been installed when you installed Breakdesigns customfieldsforall plugin. You are seeing that error with RO CSVI because with Delete custom fields relations RO CSVI deletes all custom fields relations with a product which also includes customfieldsforall. You will see this error if you create customfieldsforall field from VirtueMart. Please check with breakdesigns support on this issue.

Stockable Custom Field limited support for flields that contain characters in names: č ć š ž đ. I had field named "Veličina" (Size in english) which would skip the field on import.

In the attached debug log there is no log entries about processing of stockablecustomfields. Please check and send me latest debug log so I can check what is going on with stockablecustomfields import.

Hi Tharuna and Roland,


I have managed to build my own solution for it, without using this plugin at all. I am using external API to fetch data from provider and populate custom tables with info that I need.
Just another thing while I have you, related to a inport. I have this (default) Virtuemart products import template.
I am usually running it via cronjob, but when I run it via Administrator backend, it keeps processing records and nevers stops. What I mean by that is "processed" number keeps going up 1000,2000,3000,....9000.... and I have only 1300 products.

Again they are imported fine via Cronjob,

This is what I am getting after running a cronjob executed. I am getting JSON data back; does not matter now.

{

    "status": 200,

    "execution": {

        "output": [

            " [INFO] Processing started...",

            " [INFO] Template Example VirtueMart Product Import",

            " [INFO] Processed 1127 lines",

            " [INFO] Total Result Status",

            " [INFO] Image Incorrect: 913",

            " [INFO] Import fields Information: 1",

            " [INFO] Import fields Unsupported: 1",

            " [INFO] Manufacturer translations Updated: 1127",

            " [INFO] Medias Updated: 214",

            " [INFO] Product category relations Updated: 1127",

            " [INFO] Product media relations Added: 214",

            " [INFO] Product prices Updated: 1125",

            " [INFO] Product translations Updated: 1127",

            " [INFO] Products Updated: 1127",

            " [OK] Import has finished"

        ],

        "status": 0

    },

    "message": "Uvoz artikala je uspješno završen."

}

I am running this once a day to import any sync all products. How "heavy" is this on processing (takes 17 seconds to process)?
Sometimes I just need to "refresh" images and sometimes only few products would change price/availability.
I love component, works great so far. Just I guess it's about learning curve what settings are better and what should I avoid.

I also wonder how can handle multiple images per product, as I have not figured that one out from settings. I am storig images in folder structure

/images/virtuemart/product/__SKU__FOLDER__/sku.jpg

For each product I have dedicated folder which is named as a product's SKU and images are stored in that folder with names: sku.jpg, sku_1.jpg, sku_2.jpg, sku_3.jpg,.....

Sorry for shifting to different issue and subject.

Hello,

I am usually running it via cronjob, but when I run it via Administrator backend, it keeps processing records and nevers stops. What I mean by that is "processed" number keeps going up 1000,2000,3000,....9000.... and I have only 1300 products.

Are you downloading images on import? If you have remote images and have download option set in import template, that can take some time with import. In such cases we recommend using cron job to do the import.

I am running this once a day to import any sync all products. How "heavy" is this on processing (takes 17 seconds to process)?
Sometimes I just need to "refresh" images and sometimes only few products would change price/availability.
I love component, works great so far. Just I guess it's about learning curve what settings are better and what should I avoid.

In your import template on Image tab check if you have set Save Images on server to Yes and Redownload image to Yes. Save Images on server will download image and save it on given location. Redownload image will download images even if they exists already so that should be set to No. Check if that is the case. You can post import debug log so we can check. To get the debug log see How to collect debug information? document.

This is what I am getting after running a cronjob executed. I am getting JSON data back; does not matter now.

Response looks good. May I know how you are running the cron command to get JSON response?

I also wonder how can handle multiple images per product, as I have not figured that one out from settings. I am storig images in folder structure

/images/virtuemart/product/__SKU__FOLDER__/sku.jpg

For each product I have dedicated folder which is named as a product's SKU and images are stored in that folder with names: sku.jpg, sku_1.jpg, sku_2.jpg, sku_3.jpg,.....

Multiple images should be separated by | like image1.jpg| image2.jpg| image3.jpg. Read through Import VirtueMart images document for details.

Hi Tharuna,

Thank you for reply. To answer questions first,

Are you downloading images on import?

I am uploading them via FTP and dedicated desktop application created for this. Storing them on the server with folder structure like below. Which means I am not downloading images from anywhere, they are always sitting on the server in same location. I can always rewrite them using same.

For example my Product SKU is 993, the this is structure created for that and each other product:

/images/virtuemart/product/993/993.jpg

My curren image settings are:

Delete image relations: YES
Create image name: NO
Fill media fields: NO
Update media link only: NO
Process images: YES
Keep original file: YES
Convert image type: Do not convert
Save images on server: NO
Re-download image: YES


Only when I am adding new images to new products, I have to re-run import so that images are processed as product medial files. I can then always replace images via FTP without re-importing them.

Log files from import

I will add import debug files below for you to review them. The import that I run via joomla admin keeps running and never ends. I have to stop it manually.

May I know how you are running the cron command to get JSON response?

I have built an API based on Larave which then can kick off import by cronjob or manualy via GET request to protected endpoint. This gives me ability to run it from desktop application on single click on a button.

Multiple images import

Since I am not using CSV for the source but MySQL table, I would assume that I can set things up in a way (as you suggested):

- First colum: product_sku
- Second column: file_path

Then in "file_path" I would write value like this (without spaces): /903/903.jpg | /903/903_1.jpg | /903/903_2.jpg
Then on the import it will pickup all images, is that how this suppose to work? If that is the case I can then manage everything via deskop app. I have already created very nice drag-and-drop feature for multiple images with automatic naming. All I need is to fill sigle MySQL table with list of images for each SKU. Which is super simple and problem solved.

I will try this multi image import via template and see how it goes. Fingers crossed

Hi all,

Just to confirm that I managed to import multiple images (using Product import template) by separating them with "|". The only issue I did run into is error below.
I used default pre-installed Template "Example VirtueMart Media Product Import" which is for Product Media I assume.

Status error: 200

Status message: parsererror



Deprecated: explode(): Passing null to parameter #2 ($string) of type string is deprecated in /www/wwwroot/baby-shop.grabber.ie/plugins/csviaddon/virtuemart/com_virtuemart/model/import/mediaproduct.php on line 156



Deprecated: explode(): Passing null to parameter #2 ($string) of type string is deprecated in /www/wwwroot/baby-shop.grabber.ie/plugins/csviaddon/virtuemart/com_virtuemart/model/import/mediaproduct.php on line 186



Deprecated: explode(): Passing null to parameter #2 ($string) of type string is deprecated in /www/wwwroot/baby-shop.grabber.ie/plugins/csviaddon/virtuemart/com_virtuemart/model/import/mediaproduct.php on line 187

{"process":false,"url":"administrator\/index.php?option=com_csvi&view=logdetails&run_id=42&return=aHR0cHM6Ly9iYWJ5LXNob3AuZ3JhYmJlci5pZS9hZG1pbmlzdHJhdG9yL2luZGV4LnBocD9vcHRpb249Y29tX2Nzdmkmdmlldz10ZW1wbGF0ZXM="}

I have created separae table which has only columns: product_sku | file_url
I would then keep this MySQL table always refreshed with most recent changes OR only for files that needs to be imported and table dropped afterwards.
I am not sure about error above, I did not pull any log files for it this time. I would like to know what fields are really mandatory and what template is here best to use.
As said Product import template works but only with bunch of other fields, and it takes 60 seconds to import. Probably a lot longer when I start handling much more images. SO question is what template (if you can give me exports) and with what fields (in my case MySQL table) it will work (minimal as possible)?

Thank you!

Hello,

I will add import debug files below for you to review them. The import that I run via joomla admin keeps running and never ends. I have to stop it manually.

Thank you for the debug log. In import template on Limit tab set Lines per import to 50 or 100 and save the template. Run import and check if that makes any difference.

I used default pre-installed Template "Example VirtueMart Media Product Import" which is for Product Media I assume.

Yes the template is for product media. Also those are not errors but warnings. In import template on Details tab set Error reporting to Simple or None, save the template and run the import and check.

As said Product import template works but only with bunch of other fields, and it takes 60 seconds to import. Probably a lot longer when I start handling much more images. SO question is what template (if you can give me exports) and with what fields (in my case MySQL table) it will work (minimal as possible)?

You can use either Product import template or Product media template with file_url field to import images. Try the setting in Limit tab as said above and see if it helps.

Hi Tharuna,

Thank you for reply, I have managed to make it work, took me bit time to setup required fields. It did not work for me when I followed Guide https://rolandd.com/documentation/ro-csvi/virtuemart/import-virtuemart-images?start=4 but got it working with creating "Product" import template.
I had to add few fields in table (they are skipepd anyway but would not work without them. Interesting is that Image import was working fine BUT images would be unpublished (what I can see in Virtuemart backend). To actually publish them, I had to add field "published" in import template and set it to "1" for default value.

Got it at the and and works great.

I have another question which is Global Rules. I am trying to figure out how can I skip some products during import based on field value (from my source MySQL table OR csv).
For example I have column "completed" which I want to use for check and if it's value is "1" then product(s) should be skipped during import.

This would allow me to import single or only few products from a source table, those that I mark as as "completed" = 0.

Is this something that I can do via Rules? - I've looked at RO CSVI Conditional plugin which I have enabled but does not look like that's what I need.

Thank you!

Hello,

I had to add few fields in table (they are skipepd anyway but would not work without them. Interesting is that Image import was working fine BUT images would be unpublished (what I can see in Virtuemart backend). To actually publish them, I had to add field "published" in import template and set it to "1" for default value.

You can use published or media_published field with value set to 1. This is also explained in the document you mentioned in Adding the fields page.

For example I have column "completed" which I want to use for check and if it's value is "1" then product(s) should be skipped during import.

You can use RO CSVI Skip rule to check for value 1 and skip the record. See attached screenshot. Apply this rule on completed field in import template fields and that should do what you need.

Thank you Tharuna,

I will try that method for skipping depending on field value. What I've made in the meantime is temp sql table which holds only one (of very few) records which needs to be processed and imported. I am sending:

{

    "product_sku": "243",

    "file_url": "243/243.jpg|243/243_1.jpg|243/243_2.jpg|243/243_3.jpg|243/243_4.jpg|243/243_5.jpg"

}

Which then after accepting takes data and inserts it into temp db and kicks off import template via exec function. This is super fast and import takes only 200-300ms for that set of images.
Some magic thanks to your component and some custom work with API and app for end user.

I will try what you suggested, that will be useful for larger bulk import and skipping data that hasn't changed.

Thank you!

Hello,

What I've made in the meantime is temp sql table which holds only one (of very few) records which needs to be processed and imported.

Which then after accepting takes data and inserts it into temp db and kicks off import template via exec function. This is super fast and import takes only 200-300ms for that set of images.
Some magic thanks to your component and some custom work with API and app for end user.
That sounds interesting and processing only a few records will make import faster.

I will try what you suggested, that will be useful for larger bulk import and skipping data that hasn't changed.

Let me know on how that goes.