Categories
View Replies (22)

Resolved cron.php huge memory usage

  1. nextcomp
    nextcomp
  2. RO CSVI
  3. Tuesday, 05 July 2016
  4.  Subscribe via email
My import template works fine if I start backend manually. But not working if I start cron.php. The problem is I have got only 256MB virtual memory my web server.

/usr/local/bin/php /home/xxx/public_html/xxx/administrator/components/com_csvi/helper/cron.php --key="xxx" --template_id="8" --form.import_nolines=100 --form.import_wait=5 --file="/home/xxx/public_html/xxx/tmp/export.csv"

export.csv is about 1,5MB

I have to set --form.import_nolines=100 and --form.import_wait=5 in the backend but the cron.php is not using this. The server killed my process because 256MB is the limit!

How can I solve this problem without upgrade server memory limit?
0
Responses (22)
rolandd
rolandd
Accepted Answer Pending Moderation
Hello,

I have to set --form.import_nolines=100 and --form.import_wait=5 in the backend but the cron.php is not using this.
Cron cannot use that because it is not using any webbrowser. Cron doesn't have JavaScript.

How can I solve this problem without upgrade server memory limit?
Run multiple exports with a record limit. So let's say you run one export to do the first 10000 records and an second do the next 10000 etc.

Finally when you have your files you can run a final cron job with this command to create a single file out of it 
cat file1.csv file2.csv > totalfile.csv


A CSV file of 1.5MB is quite large so you are going to need more resources as well. The question is how much more memory does it need on your server? It's kind of weird because every line that is being exported is written directly to file.
Kind regards,

RolandD

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 1
rolandd
rolandd
Accepted Answer Pending Moderation
Hello,

Apologies, I mistook it for an export because the file is called export.csv I guess :)

How can I tell cron.php that working parts of big csv like JS?
You cannot because JavaScript is only available in your browser.

cron.php read 100 rows and than read other 100 rows etc.
You will need to break up the import file into smaller pieces in that case. CSVI can read multiple CSV files from a single folder by specifying a folder name instead of a filename to import.

Perhaps if you have PHP 7 available for CLI it may work with a single file because it handles memory quite a bit better than previous versions.
Kind regards,

RolandD

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 2
tharuna
tharuna
Accepted Answer Pending Moderation
Hello,
Yes,cron reads only one file in the folder. I am looking into this issue, will keep you posted on my findings and fixes.
Kind regards,

Tharuna

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 3
rolandd
rolandd
Accepted Answer Pending Moderation
Hello,

Attached is the patch file that will import multiple files from a folder using the cron job. You can load the patch file. After that run the cron job and all files should be imported.

Let me know how it goes.
Attachments (1)
patch_cron_20160706.zip
Kind regards,

RolandD

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 4
rolandd
rolandd
Accepted Answer Pending Moderation
Ha,

That is my mistake, this is the 6.6.0 version of the file. I am in the middle of releasing 6.6.0, so let's await that and give that a go.
Kind regards,

RolandD

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 5
nextcomp
nextcomp
Accepted Answer Pending Moderation
The export is own development. It's working fine. I use only CSVI import template. Cron.php import my export.csv. How can I tell cron.php that working parts of big csv like JS? Sorry for my English :)

cron.php read 100 rows and than read other 100 rows etc.
  1. more than a month ago
  2. RO CSVI
  3. # 6
nextcomp
nextcomp
Accepted Answer Pending Moderation
I created small files, but batch import (use folder) works only "load from server". I added correct server path (/home/xxx/public_html/xxx/tmp/import/) but Test path is failed (The given path cannot be found.) :( The path is correct!

The import works (backend), only the test path is not working.
  1. more than a month ago
  2. RO CSVI
  3. # 7
nextcomp
nextcomp
Accepted Answer Pending Moderation
Backend import is works for folder but cron.php is not :(

/usr/local/bin/php /home/xxx/public_html/xxx/administrator/components/com_csvi/helper/cron.php --key="xxx" --template_id="8"

If I add --file="/home/xxx/public_html/xxx/tmp/import/" only first csv imported too.
  1. more than a month ago
  2. RO CSVI
  3. # 8
nextcomp
nextcomp
Accepted Answer Pending Moderation
Use --help to see the help information
Processing started...
Template VM Product - Import
COM_CSVI_TOTAL_RECORDS_STATUS

Fatal error: Call to undefined method RantaiImportModel::getStatistics() in /home/xxx/public_html/xxx/administrator/components/com_csvi/helper/cron.php on line 678
  1. more than a month ago
  2. RO CSVI
  3. # 9
nextcomp
nextcomp
Accepted Answer Pending Moderation
I commented this line, works but I reached the memory limit too :( I used many small files.
  1. more than a month ago
  2. RO CSVI
  3. # 10
rolandd
rolandd
Accepted Answer Pending Moderation
Hello,

How about running multiple cronjobs? Each job processes x files of where x doesn't reach the maximum. What is the exact error anyway that you get? Are you doing any image resizing in CSVI?
Kind regards,

RolandD

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 11
nextcomp
nextcomp
Accepted Answer Pending Moderation
I create many small files, 100 rows in each files. Runs about 18 files after the server kill the process (memory limit reached).

I gave this error: [06-Jul-2016 15:45:46 Europe/Budapest] PHP Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 32 bytes) in /home/xxx/public_html/xxx/administrator/components/com_csvi/models/logs.php on line 736

In function initialiseRun() I changed $this->log->setActive(false);
  1. more than a month ago
  2. RO CSVI
  3. # 12
rolandd
rolandd
Accepted Answer Pending Moderation
In function initialiseRun() I changed $this->log->setActive(false);
If you want to turn of logging, no need to hack the code, just set the option to No in the template.

Does the import finish if the logging is turned off?
Kind regards,

RolandD

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 13
nextcomp
nextcomp
Accepted Answer Pending Moderation
No :(

Import Product VM Product - Import 2016-07-06 14:10:01 2016-07-06 14:10:08 400 No export_04.csv No debug log available

Only 4 files at this moment.

Something eating the memory.
  1. more than a month ago
  2. RO CSVI
  3. # 14
rolandd
rolandd
Accepted Answer Pending Moderation
You mean to say that at first it processed 18 files and now only 4 files? You get the same error after turning off the logging?

The problem with the memory is we don't know where the tipping point is.

Please update to version 6.6.0 and let me know the error you get with that version.
Kind regards,

RolandD

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 15
nextcomp
nextcomp
Accepted Answer Pending Moderation
(CLI cron) Import Product VM Product - Import 2016-07-06 14:43:01 2016-07-06 14:43:19 500 No No debug log available
(backend) Import Product VM Product - Import 2016-07-06 14:34:21 2016-07-06 14:40:45 Super User 6493 No Show | Open | Download

(Joomla update not working: [06-Jul-2016 16:30:17 Europe/Budapest] PHP Notice: Undefined index: extractdir in /home/xxx/public_html/xxx/administrator/components/com_installer/models/update.php on line 449)

I must reinstall to update.

"Delete after import" not working in CLI cron.
  1. more than a month ago
  2. RO CSVI
  3. # 16
rolandd
rolandd
Accepted Answer Pending Moderation
That undefined index notice is a Joomla issue, not a CSVI issue. Turning off error reporting will hide that notice and you can install as normal.

"Delete after import" not working in CLI cron.
Let's check that later.

So you can import 500 records before getting an import error? What is the exact error?
Kind regards,

RolandD

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 17
nextcomp
nextcomp
Accepted Answer Pending Moderation
Sorry for late answer I was holiday. So we have a virtual server we can use only 250MB memory. I try to work a big import file but the server security kill the process because CLI PHP use more than 250MB. You said that using small import files. I tried this, but after some small import files I reached memory limit too, server kill the CLI PHP process. After the importer finished the work one small file the CSVI program has to free all memory and start again another small file and repeat to last small file.

Best Regards,
Zsolt
  1. more than a month ago
  2. RO CSVI
  3. # 18
rolandd
rolandd
Accepted Answer Pending Moderation
Hello Zsolt,

Yeah, my thinking was wrong that smaller files will solve this problem because the CLI process is a continuous process. As for emptying the memory, I will need to find out why CSVI is using that much memory. However this is no easy task ;)

The options for you are now to create a bash script perhaps which starts PHP for each file, this way you have one PHP process per file. This may work I think. The other one is to wait until I have been able to figure things out.
Kind regards,

RolandD

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 19
rolandd
rolandd
Accepted Answer Pending Moderation
Hello,

Can you send me an email at contact@csvimproved.com with a backup of your import template and the large file you are importing? I want to see what happens here in terms of memory usage.
Kind regards,

RolandD

=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
  1. more than a month ago
  2. RO CSVI
  3. # 20
  • Page :
  • 1
  • 2


There are no replies made for this post yet.
Be one of the first to reply to this post!