1. cpointcc
  2. RO CSVI
  3. Tuesday, 26 March 2019
  4.  Subscribe via email
I am hoping you can help me with this issue today. It is a ridiculous amount of time to import such a small file.

Can you provide me with any help in making this process go faster? I have 1 import file.

Is there a way to import directly via MySQL to this? J2Store support is putting this back on you as I am using your tool.

Please advise.

I can send you credentials to the site privately if you provide me a link or email
Accepted Answer Pending Moderation
Hello,
8 hours for processing 12500 lines is way too much of time. Did you run the import manually or using cron? Setting the import to run using cron should be faster. Did you try that?

Is there a way to import directly via MySQL to this?

You cannot take values from one file to multiple tables using direct MySQL import. CSVI import processes the data and takes them to relevant tables in MySQL. You just need to find why your import is taking too much of time and a way to run it faster.
Kind regards,

Tharuna

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 1
Accepted Answer Pending Moderation
Hello,
Also, are you trying to import external images? Can you post the debug log and sample of import file to check? Importing external images could be one cause for long import hours. Debug log will tell us what is taking so long. Please post it to check. To get the debug log check How to collect debug information? document.
Kind regards,

Tharuna

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 2
Accepted Answer Pending Moderation
I will test the cron job and get back to you. Images were uploaded prior and only the field with the image path reference was imported.
  1. more than a month ago
  2. RO CSVI
  3. # 3
Accepted Answer Pending Moderation
It was a manual import. I will test a cron job today.
  1. more than a month ago
  2. RO CSVI
  3. # 4
Accepted Answer Pending Moderation
I am not seeing much improvement here. The cron job forecast is not better. I've tried 2 imports 1st ran for 2 hours - I got 3000 lines imported, then I killed the job. Deleted data from the tables and restarted with a new test and got 800 in 1/2 hour. This is totally unacceptable. I could use some help here.

Attached is the last cron log file com_csvi.log.1.zip com_csvi.log.2.zip com_csvi.log.3.zip
Attachments (3)
  1. more than a month ago
  2. RO CSVI
  3. # 5
Accepted Answer Pending Moderation
Hello,
Checking your debug log, there is a select query which takes 3 seconds for each record and so for 3000 records its 9000 seconds which comes to around 2.5 hours. We now need to check why this specific select query is taking so long. Can you run the below query directly in your phpMyAdmin and see how long it takes?

SELECT * FROM sdo2301_j2store_variants WHERE `sku` = '0000-6817-20'
Kind regards,

Tharuna

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 6
Accepted Answer Pending Moderation
This is a sample of the import file 20 lines ISI-Items-19-03-22-20-products.csv

Here is the result:

localhost/ivmgmt_sample7/sdo2301_j2store_variants/ https://whm.ivhost.org:2083/cpsess7583644161/3rdparty/phpMyAdmin/db_sql.php?db=ivmgmt_sample7
Showing rows 0 - 0 (1 total, Query took 0.0188 seconds.)

SELECT * FROM sdo2301_j2store_variants WHERE `sku` = '0000-6817-20'



38437 23266 1 0000-6817-20 Please Call for Assistance
NULL
standard 0 {"product_css_class":""} 0.00000 0.00000 0.00000 0 0.00000 0 962 0 0 0
NULL

NULL
0.00000 0 0.00000 0 0.00000 0 0
NULL
0 0
Attachments (1)
  1. more than a month ago
  2. RO CSVI
  3. # 7
Accepted Answer Pending Moderation
Hello,

Just want to add my 2 cents here. This is may be totally unacceptable to you but it is not only a CSVI issue. CSVI is dependent on how your database performs. If CSVI sends a query and gets a response after 3 seconds, there is nothing we can change codewise to speed up your database server. The issue lies in the database at that point. If the processing of some value in CSVI itself takes, 3 seconds, then we need to look at the code on how to improve that. In addition to this it might also be a case of the J2Store database structure setup not designed for mass imports. A similar issue exists for VirtueMart if you look at the VirtueMart speed up product import. A similar change may be needed for J2Store. Do know that adding indexes is not a magic trick as in, adding more and more indexes will speed things up, the wrong indexes can also slow things down.

It is almost a jigsaw puzzle that needs to be put together and not just a change of code in CSVI.

Of course we will help to try and figure this out.
Kind regards,

RolandD

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 8
Accepted Answer Pending Moderation
Well, I definitely need some help. Your link to the Speed up Product Import is for VM. Not J2Store. Doesn't really help me. I have several questions but feel like I need your help to keep moving forward on this as the project isn't going to stop since we are already vested in quite a bit of time and money to get to this point.

What is the best way to move forward with your help. the J2Store folks are putting this issue on your component. Seems someone is always pointing a finger to the other. That doesn't help me.

Below is the j2store products table and indexes. Note: No partitioning defined!
Also is the com_content for the Joomla Articles where the J2Store data is also published as well as the J2Store-products-quantity

Also, I am not getting my email notification. What email should I be whitelisting?

Please advise.
Attachments (3)
  1. more than a month ago
  2. RO CSVI
  3. # 9
Accepted Answer Pending Moderation
Hello,

Well I don't feel like we are beating around the bush here. I have given you a clear description of how the situation is and I stand by that it is not a magic fix in CSVI alone.

Your link to the Speed up Product Import is for VM. Not J2Store. Doesn't really help me.
I did say that it is for VirtueMart and only that a similar fix may be needed for J2Store. I never said this is your fix.

We have identified one issue already, that is the select Tharuna pointed out. At this moment, this takes around 2,5 hours of your import. As CSVI we just ask the database for some data and then CSVI has to wait 3 seconds before it receives an answer. A first step would be to add an index on the sku column of the sdo2301_j2store_variants table. How this is done, is explained in the tutorial I linked earlier. This has nothing to do with how CSVI works but hopefully speeds up the answer CSVI is waiting for. That is a first step we can take.

As for optimizing the database server, I have no knowledge of that as I am not a database administrator. So that is something you may need to ask your hosting partner to take a look at. Perhaps they can see some bottlenecks.

The partitions missing in your screenshots, you don't need the partitions, it is the indexes we are looking at.

Be aware, once the import is up-to-speed, your site can be slow because of the queries done by the extensions you use for retrieving the stored data. You may need to do optimizing in that area as well. That is just a heads up not saying that it will happen but could happen.

Also, I am not getting my email notification. What email should I be whitelisting?
Emails are coming from contact@csvimproved.com
Kind regards,

RolandD

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 10
Accepted Answer Pending Moderation
Hi Roland,

I did run a new cron job yesterday. Of course the log files are only showing a portion of the import. I have no idea how long it really took to import. The dashboard shows:

Action Action type Template name Start End User Records Run cancelled Debug log
Import Product Product File Import Optimized 01-04-2019 16:49:01 01-04-2019 16:49:02 12489 No Show | Open | Download
Import Product Product File Import Optimized 28-03-2019 10:58:01 28-03-2019 10:58:01 12489 No Show | Open | Download
Import Product Product File Import Optimized 27-03-2019 10:58:01 27-03-2019 10:58:01 12489 No Show | Open | Download

I know the 3/27/19 import took 8 hours because I was here watching it on the server. Last night I let it run.

I did index the SKU column of the sdo2301_j2store_variants table. j2store-variants-table-index-screenshot.jpg

So with that new information, is there a way to track the actual import time?

I will look further into the site tables and optimization.

It is our server, I will talk to the server manager on this issue.

Any other information that you can provide to help with this process is greatly appreciated.
Attachments (1)
  1. more than a month ago
  2. RO CSVI
  3. # 11
Accepted Answer Pending Moderation
Hello,
So with that new information, is there a way to track the actual import time?

We need to check your latest debug log. You can increase Number of lines in log file from CSVI Dashboard >> Click Options >> Site tab and run the import again. Debug log will also tell us if the time has improved with the select query which was taking 3 seconds previously.

Any other information that you can provide to help with this process is greatly appreciated.

Can you also post your import file? We will run the import on our local server and check if we can reproduce your issue. You can send it as an email or in a private message.
Kind regards,

Tharuna

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 12
Accepted Answer Pending Moderation
I have attached the log file. It is inconclusive. As far as sending the import file, the current has proprietary info in it. When I get the next import that will exclude that I will send it. It may not be today.

What email do I send it too? I don't see where to connect via private message.
Attachments (1)
  1. more than a month ago
  2. RO CSVI
  3. # 13
Accepted Answer Pending Moderation
Hello,

The logfile at least points out that the index on that SKU field didn't do anything because we are still waiting for 3 seconds to get an answer. What I do spot now in your screenshot is that your SKU index is 191 characters long. I am guessing you don't have SKUs that are 191 characters long ;) Reducing this number to a value that is more representative for your SKUs, perhaps something like 25 maybe, may help. Simply because there is less data to index and keep track off and you can get a faster hit.

It would be helpful if the server people can give any clue why this takes so long.

One other thing I noticed in the log is that you are resizing thumbnails, it may sometimes be useful to first run the product import and the image resizing in a separate import. This way you will have all the product info in place and the pictures come after that.

As far as sending the import file, the current has proprietary info in it.
Understood, no problem.

When I get the next import that will exclude that I will send it. It may not be today.
Thank you.

I don't see where to connect via private message.
You can click on our name to see our profile and there will be a button to send a private message. Alternatively you can also click on your own profile at the top right corner where there is a picture and it will have a link to the Private Messages area. To email, you can use contact@csvimproved.com.
Kind regards,

RolandD

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 14
Accepted Answer Pending Moderation
Our server staff is asking if you have any specific optimization settings that are optimal for your program. The MySQL has not been optimized, but any input you have would be helpful. Its a fairly new server and tweaking it to fit this project is a new task. I did however optimize all the tables in the database this am.

No other clients have had issues with the server settings at this point until now with this project. If your product is completely dependent on the server optimization we need some criteria to start with.

I have changed the char count to 40 on the sku in the j2store_variants table.

The images: I am not importing images. We are only referencing the path in the import. They are already pre-sized. How or what can I change to avoid this?
  1. more than a month ago
  2. RO CSVI
  3. # 15
Accepted Answer Pending Moderation
Hello,

If your product is completely dependent on the server optimization we need some criteria to start with.

We have no special requirements for the database server to run CSVI. When we send a request and get an answer 3 seconds later it might be because the database server doesn't have enough resources to process it faster. As I mentioned before, I am not a database admin, so I don't know how to fine-tune a database server.

No other clients have had issues with the server settings at this point until now with this project.
Are they all importing the same amount of data in the same setup using J2Store?

How or what can I change to avoid this?
In your template go to the Images tab and turn off the Create thumbnails, because that is now turned on. That should make a difference.

I was wondering, did you have the debug log enabled when you ran the full import? The debug log also takes time, so having that disabled when doing the full run will also make a difference.
Kind regards,

RolandD

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 16
Accepted Answer Pending Moderation
OK we will continue to troubleshoot the server optimization.

Cannot compare client projects, just no issues in general. This project are only one using J2Store.

Images. there is no thumnail generating option.
Create image name set to no,
Image name based on
Image name format -JPG
Process images - No

Debug is not turned on. Only enable logging. I needed to see this for error troubleshooting.

One main issue I have just discovered. In my import I have a Product ID. That ID must stay the same. I don't see anywhere where it is being added. It looks like the product ID is being auto incremented. THIS IS A MUST. How can I insure the integrity of the Product ID to be based on the import file data?
  1. more than a month ago
  2. RO CSVI
  3. # 17
Accepted Answer Pending Moderation
Hello,

Images. there is no thumnail generating option.
I see, it is hidden because the Process images is set to No. This is an issue because we shouldn't be creating the thumbnails. We will fix this and supply a patch once fixed.

Debug is not turned on. Only enable logging. I needed to see this for error troubleshooting.
A little miscommunication, the logging is what we call the debug log. Of course this is turned on as we are testing but when doing a live run, when this is turned off, the import will be faster too as it has a lot less of data to write. That is the only thing I was pointing out.

In my import I have a Product ID. That ID must stay the same.
Which ID is this? Is this your own ID?

I don't see anywhere where it is being added.
Have you mapped this field to a J2Store field?

It looks like the product ID is being auto incremented.
Correct, this is the J2Store database schema as most tables always have a primary key which is auto-incremental to ensure you can target a specific entry. This is the J2Store internal ID and is managed by the database.

How can I insure the integrity of the Product ID to be based on the import file data?
That is what we use the SKU for. An SKU must be unique as that is what the name is as well. Stock Keeping Unit, an identifier that identifies a specific product. Based on the SKU we will check if the product exists, so yes we will update the product, if it doesn't exist we create the product (unless you have set the option not to create new products).

Let me know if you have any further questions.
Kind regards,

RolandD

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 18
Accepted Answer Pending Moderation
Hello,
Attached is the patch file mentioned by Roland for thumbnails not to be created when Process image is set to No. Load the patch file and you should not be seeing any more image processing in your import.
Attachments (1)
Kind regards,

Tharuna

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 19
Accepted Answer Pending Moderation
Images. there is no thumbnail generating option.

I see it is hidden because the Process images are set to No. This is an issue because we shouldn't be creating the thumbnails. We will fix this and supply a patch once fixed.
Debug is not turned on. Only enable logging. I needed to see this for error troubleshooting. | I have uploaded the patch file. I will see if that improves anything. Regarding the rest:

A little miscommunication, the logging is what we call the debug log. Of, course this is turned on as we are testing but when doing a live run, when this is turned off, the import will be faster too as it has a lot less of data to write. That is the only thing I was pointing out In my import I have a Product ID. That ID must stay the same.

Which ID is this? Is this your own ID?
I don't see anywhere where it is being added. Yes this is our product ID. I want to keep it the same. It is in the import file in the first field

Have you mapped this field to a J2Store field? It looks like the product ID is being auto incremented. See attached: field-list-for-template.jpg

Correct, this is the J2Store database schema as most tables always have a primary key which is auto-incremental to ensure you can target a specific entry. This is the J2Store internal ID and is managed by the database.
How can I insure the integrity of the Product ID to be based on the import file data?

That is what we use the SKU for. An SKU must be unique as that is what the name is as well. Stock Keeping Unit, an identifier that identifies a specific product. Based on the SKU we will check if the product exists, so yes we will update the product, if it doesn't exist we create the product (unless you have set the option not to create new products). Our SKU is in.
Attachments (1)
  1. more than a month ago
  2. RO CSVI
  3. # 20
  • Page :
  • 1
  • 2
  • 3
  • 4


There are no replies made for this post yet.
Be one of the first to reply to this post!