1. Zitelia
  2. RO CSVI
  3. Monday, 14 March 2016
  4.  Subscribe via email
Hello,

I´m triying to import a field with 10000 lines to prove, because the original have 400.000 lines.

My problem is that when I´m have selected the archive with the 5000 lines and I clic in preview appears like in image(grey)
and I can start the import but with a archive with 3000 lines, it works.

The template limits is configurate like in image (limits)

I´m running the web in my localhost for this testing imports before install all in production.

So how I can import a testing field of 10000 lines to see how much time takes to import???

Thanks
Accepted Answer Pending Moderation
Hello,

The problem seems to be that your local machine can't handle the large file in a timely matter. You will have to increase the maximum execution time I think to be able to view the preview.

So how I can import a testing field of 10000 lines to see how much time takes to import???
That I think is useless since comparing your local machine to a live server is like doing a race between a Fiat 500 and a Ferrari :) It would be better to run the tests on a test website on your server. On your local machine it should be enough to test with a small file.

The way you have entered the Lines per import seems to indicate you haven't read the page Manage timeouts yet. These limits are only used during the import, not the preview.
Kind regards,

RolandD

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 1
Accepted Answer Pending Moderation
Thank you very much Rolandd,
I had thought in the problem of my localhost like you said, just after write here the problem but it´s perfect that you confirm me ;)

Thanks
  1. more than a month ago
  2. RO CSVI
  3. # 2
Accepted Answer Pending Moderation
Hello,

after change the configuration of the local host, read the documentation and view this error message

[18-Mar-2016 10:52:21 Europe/Paris] PHP 16. file_get_contents() D:\Servidores\wamp 2.5\www\edicionestantin3\administrator\components\com_csvi\helper\source.php:380

I´ve seen that the command of the line 380 ($sourcefile = file_get_contents($urlfile);) load in memory all the archive that I want to import. This archive have 500MB, and that collapse the memory.

Whats is the solution for this??

Thanks
  1. more than a month ago
  2. RO CSVI
  3. # 3
Accepted Answer Pending Moderation
Hello,

Can you give the attached patch file a try? You can load the patch file and after that run your import.

Let me know how it goes.
Attachments (1)
Kind regards,

RolandD

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 4
Accepted Answer Pending Moderation
Hello,

With this patch now the option for import with external url don´t recognize the archive, and is a txt. The Import from de server works with the patch.

I let the computer Importing entire weekend and the problem now is that there are 400.000 products with img,and the memory of the server were full, so we have to: (2 options)
+ separate import process for that because it takes quite some more time to import products with img (one process to products and other for images)

+ or if is possible that we can do in the same import both, products and images, but in this case the image don´t download to the server but show the image with external url??

Thanks
  1. more than a month ago
  2. RO CSVI
  3. # 5
Accepted Answer Pending Moderation
Hello,

With this patch now the option for import with external url don´t recognize the archive, and is a txt.
Make sure you have set in your template to treat it as a CSV file. This patch makes no changes in that regard.

the problem now is that there are 400.000 products with img,and the memory of the server were full
If you are using the browser to import the products make sure you setup your template according to the instructions in the Manage timeout document. This will refresh everything after x records as set in the template.

or if is possible that we can do in the same import both, products and images, but in this case the image don´t download to the server but show the image with external url??
You can import images with the products. CSVI can also download external images to the server if you have set in your template.

You will need to provide the information as requested at the top of the page under Looking for a useful answer?. Especially the debug log. Just saying "it doesn't work" makes it impossible to help you.
Kind regards,

RolandD

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 6
Accepted Answer Pending Moderation
Hello Rolandd

the path archives and the configuration of limit times fix the problem loading from server, like I said you last time, but import from URL still have problems now when I take an import from the URL, I don´t have errors in PHPlog and I get a White screen after 3minutes in preview step of the import (I don´t see a preview of the import).

How can I solve it? what information do you need?

the URL for the archive that I want to import is : http://www.zitelia.it/servicios_web/articulos.php?fr_usuario=zzzz&fr_clave=66666s6
  1. more than a month ago
  2. RO CSVI
  3. # 7
Accepted Answer Pending Moderation
Hello,

but import from URL still have problems now when I take an import from the URL, I don´t have errors in PHPlog and I get a White screen after 3minutes in preview step of the import (I don´t see a preview of the import).
Did you try to download that file? I did download that file and it is a huge file, with my 500Mbit connection the download still takes nearly 10 minutes. This is with several tries because the server closed the connection. This will be the reason why you get a white screen, the download of the file simply takes too long. It is a 364MB file.

The only way around this is to have a script that will take care of the download and then let CSVI import your file from the local server. A file that size is currently not going to be downloaded fast enough to let the import start.
Kind regards,

RolandD

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 8
Accepted Answer Pending Moderation
Hi Roland,

As you told us you test the CSVI to import 1 million of products in 40 minutes, so we think that we can do it in less time because our file has 400.000 products.
I have to import that file and it takes a lot of time to end (more or less 30 products in 5 seconds so it will be end in 18 hours). How config is the best in the CSVI to import that file in few time?

Thanks.
  1. more than a month ago
  2. RO CSVI
  3. # 9
Accepted Answer Pending Moderation
Hello,

As you told us you test the CSVI to import 1 million of products in 40 minutes, so we think that we can do it in less time because our file has 400.000 products.
This also depends on your server configuration of course and how it is fine-tuned. That is up to the system administrator to fine-tune the server. I assume you do have a dedicated server as well.

How config is the best in the CSVI to import that file in few time?
Once change you need to make is to add an index on the product_sku field in the virtuemart_products table.

The other thing is, don't do image processing when importing the products, do that in a separate import. The image processing takes a lot of time as well.
Kind regards,

RolandD

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 10
Accepted Answer Pending Moderation
Hello Rolandd,

Doing the adding an index on the product_sku field in the virtuemart_products table improves the efficiency
but after a while the response time server increases from 3 to 25 sec

How can I improve it?
  1. more than a month ago
  2. RO CSVI
  3. # 11
Accepted Answer Pending Moderation
Hello,

after a while the response time server increases from 3 to 25 sec
You mean during the import or loading the site itself?

I am not sure which response time you are talking about.

Do you know, you may need to tweak MySQL settings as well but the database administrator will know what to do with that. This is out of my expertise as well.
Kind regards,

RolandD

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 12
Accepted Answer Pending Moderation
Hello,

The time of response of the server during the import that I see in the import view page

this is the time that increase from 3 sec at the beginning to 25 sec after a while
  1. more than a month ago
  2. RO CSVI
  3. # 13
Accepted Answer Pending Moderation
Hello,

That response depends on your server, apparently your server starts to slow down as more products are imported. What needs to be checked is where your MySQL server is slowing down and how it's configuration can be optimized. It will be good to use the latest available version of MySQL as this is quite a lot faster than the older versions.

In terms of storage, of course the use of SSD versus HDD drives will make a difference as well.

You are reaching the point where CSVI is able to send so many database queries that it gets into a queue at the database.
Kind regards,

RolandD

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 14
  • Page :
  • 1


There are no replies made for this post yet.
Be one of the first to reply to this post!