Categories
View Replies (5)

Resolved Performance related recommendations

  1. konstantinos29
    konstantinos29
  2. RO CSVI
  3. Monday, 03 August 2020
  4.  Subscribe via email
Hello,

We need your advice in an issue that is only partially related to RO CSVi, but maybe you can help.

Wometimes when we do mass CSV imports/exports, the whole frontend slows to a crawl. It also creates an issue with new VM orders appearing blank, without customer details, which is a major problem.

To resolve the problem, we need to do a server reboot.

We use a managed VPS service with 8GB RAM. We have set php memory limit to 1024MB (we tried 512MB but some CSV imports/exports fail with a timeout).

Do you recommend we increase the php limit to more than 1024MB?
Is there a way to not let CSVi saturate the server capacity so aggressively? (Export templates do not have an option to chunk the export process to let the server "breathe" )

When we override server limits in the template, does this affect only CSVi process running at that moment? Do the limits change back to server defaults after the process is finished?

Any other advice appreciated. Thank you
0
Accepted Answer
tharuna
tharuna
Accepted Answer Pending Moderation
Hello,
Scenario 1 is fine, you can reduce the time between imports to 5 from 10.

(btw is that in seconds?

Yes, time in seconds.

what exactly is the benefit of that for the server?)

It lets server to delay the execution of import for the seconds set in time between imports. So if you are importing 100 records in a batch, after one batch is processed server halts for 5 seconds and then executes the next batch. This is basically used to reduce the load on server.
Kind regards,

Tharuna

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # Permalink
Responses (5)
rolandd
rolandd
Accepted Answer Pending Moderation
Hello Konstantin,

Do you recommend we increase the php limit to more than 1024MB?
Have you identified the memory to be a problem? Importing CSV files shouldn't need that much memory.

Is there a way to not let CSVi saturate the server capacity so aggressively?
RO CSVI is a PHP program and as such is a PHP process on the server, if you run the process as a cronjob, you can use a program like nice that prevents programs to use too many server resources.

When we override server limits in the template, does this affect only CSVi process running at that moment?
This only affects that current process as they are runtime changes. Once the run is finished, everything related to it is gone as well.

One thing you did not mention, did you check the database server? Often if you have really large imports, the process can also stall due to waiting for the database. Optimizing the database can be useful as well. Just don't ask me which settings :D That is out of my expertise.

Are you running the imports/exports as cronjobs?
Kind regards,

RolandD

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 1
konstantinos29
konstantinos29
Accepted Answer Pending Moderation
Have you identified the memory to be a problem? Importing CSV files shouldn't need that much memory.


Memory is part of the problem, in the sense that anything much below 1024MB, results in import/export timeouts (our timeout limit is 2000 seconds if I remember correctly).

We run imports/exports from the Joomla Component. We are doing them supervised (not scheduled), so we didn't see a need to do as cronjobs.
Unless there is a good reason to do so.

Our database server is definitely stressed as well (60k+ products, custom field tables in excess of 300MB). But we have 0 configuration control over that. We can only configure the basic PHP/nginx parameters.

Thank you for your time.
  1. more than a month ago
  2. RO CSVI
  3. # 2
rolandd
rolandd
Accepted Answer Pending Moderation
Hello,

Since you have no control over the database, that is a pity as it can be a good performance gain as well. The amount of 60k+ products won't make the database sweat but the custom fields definitely add a layer of complexity and resource using to it.

our timeout limit is 2000 seconds if I remember correctly
This is not the golden egg of course, you can lower the timeout and lower the number of records to process per run. This way, the PHP process is ended and a new one started. This also means, garbage cleanup, so the memory gets freed as well. Have you tried that? The goal doesn't have to be to import all in 1 run but the goal is to import it all.

As for the export, there is no batching there as it hasn't been really needed in general. Most servers just keep serving data until the stream ends.

Unless there is a good reason to do so.
You don't have the limits as you have them in the browser. That is generally one of the reasons to run large imports/exports.
Kind regards,

RolandD

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 3
konstantinos29
konstantinos29
Accepted Answer Pending Moderation
Thanks,

one more question...

For our imports, we have a choice on how many products a CSV file can contain.

We can have 2 scenarios.

1) One large CSV with 10000 products, set the limits in CSVi, to lines per import to 500, and add time between imports to 10 (btw is that in seconds? what exactly is the benefit of that for the server?)

2) Many small CSV product files, 500 products per file, no limits on CSVi template.

We prefer to work with scenario 1, because we will end up with fewer and more manageable CSV files.
  1. more than a month ago
  2. RO CSVI
  3. # 4
  • Page :
  • 1


There are no replies made for this post yet.
Be one of the first to reply to this post!