1. mrblonde
  2. RO CSVI
  3. Thursday, 22 June 2017
  4.  Subscribe via email
Hi, since upgrading to CSVI Pro 7.2.3 (from 7.1.0) in Joomla 3.7.2 and PHP 7 I'm having an issue with importing a full CSV of 5021 products. After 31 seconds it gives a 500 error. I've once successfully imported this same CSV with no issue but as was only a test the imported products/categories were deleted so I know the VirtueMart import template is OK. Before the 500 error it does successfully import about 30 products with no problem.

ERROR LOG
[Thu Jun 22 17:17:12 2017] [warn] [client xxx.xxx.xxx.xxx] mod_fcgid: read data timeout in 31 seconds, referer: http://xxx/administrator/index.php?option=com_csvi&view=import
[Thu Jun 22 17:17:12 2017] [error] [client xxx.xxx.xxx.xxx] Premature end of script headers: rantai.php, referer: http://xxx/administrator/index.php?option=com_csvi&view=import

ACCESS LOG (most recent prior to downloading CSVI log)
- - [22/Jun/2017:17:16:22 +1200] "GET /media/jui/css/chosen.css?8a6f23c2c9ec8ee7b61e4fa4b1015d13 HTTP/1.1" 200 2407 "http://xxx/administrator/index.php?option=com_csvi&view=imports" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:54.0) Gecko/20100101 Firefox/54.0"
- - [22/Jun/2017:17:16:22 +1200] "GET /media/jui/js/chosen.jquery.min.js?8a6f23c2c9ec8ee7b61e4fa4b1015d13 HTTP/1.1" 200 7609 "http://xxx/administrator/index.php?option=com_csvi&view=imports" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:54.0) Gecko/20100101 Firefox/54.0"
- - [22/Jun/2017:17:16:22 +1200] "GET /administrator/components/com_csvi/assets/css/display.css?8a6f23c2c9ec8ee7b61e4fa4b1015d13 HTTP/1.1" 200 1973 "http://xxx/administrator/index.php?option=com_csvi&view=imports" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:54.0) Gecko/20100101 Firefox/54.0"
- - [22/Jun/2017:17:16:22 +1200] "GET /administrator/components/com_csvi/assets/js/csvi.js?8a6f23c2c9ec8ee7b61e4fa4b1015d13 HTTP/1.1" 200 5780 "http://xxx/administrator/index.php?option=com_csvi&view=imports" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:54.0) Gecko/20100101 Firefox/54.0"
- - [22/Jun/2017:17:16:24 +1200] "POST /administrator/index.php?option=com_csvi&view=imports HTTP/1.1" 303 20 "http://xxx/administrator/index.php?option=com_csvi&view=imports" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:54.0) Gecko/20100101 Firefox/54.0"
- - [22/Jun/2017:17:16:24 +1200] "GET /administrator/index.php?option=com_csvi&task=importsource.source&runId=109 HTTP/1.1" 200 6135 "http://xxx/administrator/index.php?option=com_csvi&view=imports" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:54.0) Gecko/20100101 Firefox/54.0"
- - [22/Jun/2017:17:16:36 +1200] "POST /administrator/index.php?option=com_csvi&view=importsource HTTP/1.1" 303 20 "http://xxx/administrator/index.php?option=com_csvi&task=importsource.source&runId=109" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:54.0) Gecko/20100101 Firefox/54.0"
- - [22/Jun/2017:17:16:38 +1200] "GET /administrator/index.php?option=com_csvi&task=importpreview.preview&runId=109 HTTP/1.1" 200 7537 "http://xxx/administrator/index.php?option=com_csvi&task=importsource.source&runId=109" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:54.0) Gecko/20100101 Firefox/54.0"
- - [22/Jun/2017:17:16:41 +1200] "POST /administrator/index.php?option=com_csvi&view=import HTTP/1.1" 200 4273 "http://xxx/administrator/index.php?option=com_csvi&task=importpreview.preview&runId=109" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:54.0) Gecko/20100101 Firefox/54.0"
- - [22/Jun/2017:17:16:41 +1200] "GET /administrator/components/com_csvi/assets/js/jquery.timers.js?8a6f23c2c9ec8ee7b61e4fa4b1015d13 HTTP/1.1" 200 1131 "http://xxx/administrator/index.php?option=com_csvi&view=import" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:54.0) Gecko/20100101 Firefox/54.0"
- - [22/Jun/2017:17:16:18 +1200] "GET /administrator/index.php?option=com_installer&view=update&task=update.ajax&a07f75edd24232dd1f307f86d6874e7e=1&eid=0&skip=700 HTTP/1.1" 200 22 "http://xxx/administrator/index.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:54.0) Gecko/20100101 Firefox/54.0"
- - [22/Jun/2017:17:16:41 +1200] "GET /administrator/components/com_csvi/rantai/rantai.php?task=import&runId=109&_=1498108601561 HTTP/1.1" 500 339 "http://xxx/administrator/index.php?option=com_csvi&view=import" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:54.0) Gecko/20100101 Firefox/54.0"
- - [22/Jun/2017:17:17:20 +1200] "POST /administrator/index.php?option=com_csvi&view=imports HTTP/1.1" 303 20 "http://xxx/administrator/index.php?option=com_csvi&view=import" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:54.0) Gecko/20100101 Firefox/54.0"
- - [22/Jun/2017:17:17:20 +1200] "GET /administrator/index.php?option=com_csvi&view=imports HTTP/1.1" 200 6869 "http://xxx/administrator/index.php?option=com_csvi&view=import" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:54.0) Gecko/20100101 Firefox/54.0"
- - [22/Jun/2017:17:17:23 +1200] "GET /administrator/index.php?option=com_csvi&view=logs HTTP/1.1" 200 10401 "http://xxx/administrator/index.php?option=com_csvi&view=imports" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:54.0) Gecko/20100101 Firefox/54.0"
- - [22/Jun/2017:17:17:23 +1200] "GET /media/jui/css/jquery.searchtools.css?8a6f23c2c9ec8ee7b61e4fa4b1015d13 HTTP/1.1" 200 630 "http://xxx/administrator/index.php?option=com_csvi&view=logs" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:54.0) Gecko/20100101 Firefox/54.0"
- - [22/Jun/2017:17:17:23 +1200] "GET /media/jui/js/jquery.searchtools.min.js?8a6f23c2c9ec8ee7b61e4fa4b1015d13 HTTP/1.1" 200 1794 "http://xxx/administrator/index.php?option=com_csvi&view=logs" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:54.0) Gecko/20100101 Firefox/54.0"
- - [22/Jun/2017:17:17:23 +1200] "GET /media/system/js/html5fallback.js?8a6f23c2c9ec8ee7b61e4fa4b1015d13 HTTP/1.1" 200 2344 "http://xxx/administrator/index.php?option=com_csvi&view=logs" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:54.0) Gecko/20100101 Firefox/54.0"
- - [22/Jun/2017:17:17:28 +1200] "GET /administrator/index.php?option=com_csvi&task=logs.downloadDebug&run_id=133 HTTP/1.1" 200 28531 "http://xxx/administrator/index.php?option=com_csvi&view=logs" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:54.0) Gecko/20100101 Firefox/54.0"
- - [22/Jun/2017:17:23:03 +1200] "GET /administrator/index.php?option=com_csvi&view=about HTTP/1.1" 200 7499 "http://xxx/administrator/index.php?option=com_csvi&view=logs" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:54.0) Gecko/20100101 Firefox/54.0"

Attached is the CSVI generated log and CSV though for commercial sensitivity reasons have stripped the RRP and Price fields out.

All folders are writable (in CSVI About tab) and have successfully imported another CSV of another 4000+ products from client’s old site but the image importing was from a local folder.

Would appreciate any advice you can give. It seems to work OK but times out after 31 seconds consistently. I did try changing some limits in the import template: Lines per import = 200, Time between imports = 1, Override system limits = No. To the best of my knowledge nothing has changed on the host server that could be causing it.

Regards,
Julian.
Accepted Answer Pending Moderation
Hi Tharuna,

Thanks for your suggestion of skipping the category_path field. That had crossed my mind but thought I'd check incase there was an option to ignore existing categories but skipping should be OK as at any given time there is likely only going to be a few new products if any which can be manually categorised.

Again thank you so much for your support.

Regards,
Julian.
  1. more than a month ago
  2. RO CSVI
  3. # 1
Accepted Answer Pending Moderation
Hello Julian,
Yes changing the lines per import to 5 does allow it to finish though obviously is a very slow process to import over 5000 products. Hopefully the vendor is able to supply images so this can be done locally.

Yes it would be a slow process but it would have worked because once the number of lines per import is reached the script restarts and the server memory is also restarted and released.

I know the category path could be updated in the CSV but we're hoping to automate future updates from the vendor but can't do this if the category structure isn't kept.

Is it a must that you need to update your category_path in your next import? How about using skip field in place of category_path in your import? CSVI will update the category when it finds it in the import fields. You can always avoid it using skip field but you should also remember that any new categories will not be updated.
Kind regards,

Tharuna

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 2
Accepted Answer Pending Moderation
Hi,

Yes changing the lines per import to 5 does allow it to finish though obviously is a very slow process to import over 5000 products. Hopefully the vendor is able to supply images so this can be done locally.

Once products are imported with the flat list of categories from the CSV, we nest each supplied category under parent categories we create within VirtueMart which the vendor doesn't supply in their CSV. How can we get CSVI to keep the category path on the server without future imports un-doing the subcategory structure?

For example if a "ASUS Xonar DX 7.1 Channel PCIe Audio Card" is saved in the CSV with the category "Sound Cards". We move Sound Cards to be a subcategory of "Hardware" which we created in VirtueMart. On the next import all sound cards are added to a new top level category Sound Cards and the subcategory is then empty.

I know the category path could be updated in the CSV but we're hoping to automate future updates from the vendor but can't do this if the category structure isn't kept.

I appreciate your help.

Regards,
Julian.
  1. more than a month ago
  2. RO CSVI
  3. # 3
Accepted Answer Pending Moderation
Hello Julian,
There is nothing wrong i could see in your debug logs but that the number of lines to import have increased to 540 compared to 160 from your last try. What if you set number of lines per import to 5 instead of 20?
Kind regards,

Tharuna

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 4
Accepted Answer Pending Moderation
Hi,

As suggested I enabled image processing and set Resize Large Image to No. It got to 520 processed when it gave the 500 error again with Lines per import as 20 and Time between imports as 10 like last time.

I've attached the log just incase it varies from last time.

Regards,
Julian.
Attachments (2)
  1. more than a month ago
  2. RO CSVI
  3. # 5
Accepted Answer Pending Moderation
Hello Julian,
I disabled image processing on that import template, started the import again and it successfully finished so something definitely points to downloading the image from a remote server. I heard back from the vendor and they're not aware of any throttling or limiting at their end.

Limiting should be on your server and not on vendor as the image is downloaded on your server. Now that we know that the issue is because of image processing, let us try something. Looking at your debug log, i see that you have set the option Resize large image to Yes, Can you set Process images to Yes and set Resize large image to No and then try the import again?

To get around this problem I've asked whether their images are downloadable from a single location e.g. via FTP that I can upload to our server and run the import using local images but haven't had a reply yet.

Yes, that would make your import run faster by having images on local server.
Kind regards,

Tharuna

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 6
Accepted Answer Pending Moderation
Hi Tharuna,

Sorry for my delay in reply.

I disabled image processing on that import template, started the import again and it successfully finished so something definitely points to downloading the image from a remote server. I heard back from the vendor and they're not aware of any throttling or limiting at their end.

To get around this problem I've asked whether their images are downloadable from a single location e.g. via FTP that I can upload to our server and run the import using local images but haven't had a reply yet.

Regards,
Julian.
  1. more than a month ago
  2. RO CSVI
  3. # 7
Accepted Answer Pending Moderation
Hello Julian,
Yes, sometimes downloading images from another server takes time and it possible to hit the maximum server limit with it. What if you try the import without images? Does the import finish then?
Kind regards,

Tharuna

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 8
Accepted Answer Pending Moderation
Hi Tharuna,

Changing the limit and and time between values made a small difference but after a couple of cycles (60 products imported) it gave the same error. I noticed the Last Server Response time started to inch over so dropped it back to 25. It went through three cycles (75 products imported) then failed again.

I played around and set the Lines per import to 20 and increased time between imports to 10. It went through 8 cycles (160 products) then as other times the Last Server Response time just kept ticking up to 30 then failed.

The CSV is loading product images from a totally different external server which is physically located in the same country as ours so server to server response/ping time shouldn't be an issue. I've randomly checked a dozen images and they all appear to be very low resolution with a width of only 300px and are between 90kB to 180kB in size. Just incase it is adding to the load somehow I turned off the import template's resize main image option which isn't really needed in CSVI but still no difference.

I'm wondering whether the remote server has some form of throttling which could be causing it. I attempted to contact a technical person I have details for yesterday but they haven't got back to me.

Regards,
Julian.
  1. more than a month ago
  2. RO CSVI
  3. # 9
Accepted Answer Pending Moderation
Hello Julian,
Since you are saying that you are able to import 30 products before hitting server error, it is possible that you are reaching the maximum server limit. What if you try to set number of lines to import to 30 and Time between imports to 5 and then run the import?

All folders are writable (in CSVI About tab) and have successfully imported another CSV of another 4000+ products from client’s old site but the image importing was from a local folder.

Is this import done on the same site and same server or on a different server?
Kind regards,

Tharuna

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 10
  • Page :
  • 1


There are no replies made for this post yet.
Be one of the first to reply to this post!