1. kittmaster
  2. RO CSVI
  3. Sunday, 20 September 2015
  4.  Subscribe via email
I have over 90K import to redo, at first all is well, but as I move to larger imports, it crashes after import preview.

I've tested 8700 import lines ok, went to 26000 lines and it causes it to crash, no error, no nothing....just white screen and stops.

Any ideas?

I've added the file here but is password protect as it now has private data I don't want people viewing.

Send me a private message and I'll give the password of the .csv file

Thanks Roland.

Chris
Accepted Answer Pending Moderation
Using Chrome.... :whistle:
  1. more than a month ago
  2. RO CSVI
  3. # 21
Accepted Answer Pending Moderation
Works fine in Chrome for me as well :evil:
  1. more than a month ago
  2. RO CSVI
  3. # 22
Accepted Answer Pending Moderation
hmmm......let me give it a go again......see what happens
  1. more than a month ago
  2. RO CSVI
  3. # 23
Accepted Answer Pending Moderation
ummmmmm yeah.......seeing it work now here as well......the video I shot was what it was doing from day 1......strange......and now I see that the .csv file is not being deleted anymore either.......what gives there?

Now it automagically works.......and you didn't change a single thing?
  1. more than a month ago
  2. RO CSVI
  3. # 24
Accepted Answer Pending Moderation
I didn't change anything, my first step was to reproduce what your video shows. I didn't get there because it just worked. Perhaps a change on the server?
  1. more than a month ago
  2. RO CSVI
  3. # 25
Accepted Answer Pending Moderation
Nothing on my end.....in the least.....I'm using 1and1.....they suck in updates......but would be really strange to have change during this problem set.....

Well, whatever, seems to be working now, if it happens again, I'll give a shout....

Just as an FYI, only the file size max upload limit would determine the import size I can put up there right? If 26K is 15MB, I suspect I could do a half and half to all 90k done within two files sets?

There are no hard limits in the code is I guess what I'm asking for max import line items?
  1. more than a month ago
  2. RO CSVI
  3. # 26
Accepted Answer Pending Moderation
There are no hard limits in the code is I guess what I'm asking for max import line items?
No hard limits. The maximum lines per batch can be configured in the template to work around the maximum execution time.

I suspect I could do a half and half to all 90k done within two files sets?
You can do that or FTP the full file and have CSVI read that one. In that case the file upload limit is not in play.
  1. more than a month ago
  2. RO CSVI
  3. # 27
Accepted Answer Pending Moderation
Ok, tried that theory and updated....I merged all the data....and went to do the preview again:

test.aaron-beachproductions/tmp/Master-All.csv is not a readable file

It has since deleted the file again like the video.

ARGGGGGGGG
  1. more than a month ago
  2. RO CSVI
  3. # 28
Accepted Answer Pending Moderation
That is very odd as we both have seen it working before as well. Perhaps something on the server gets triggered? CPU usage for example?

I did release CSVI 6.2.0 just now, it wouldn't harm to at least update to that release and see if it makes any difference.
  1. more than a month ago
  2. RO CSVI
  3. # 29
Accepted Answer Pending Moderation
6.2.0 Same thing, eats the file and says test.aaron-beachproductions/tmp/Master-All.csv is not a readable file

EDIT: It didn't eat the file, just error.

I went and did it again, see 90K lines....did it again......90K lines.....there is some oddity there.
  1. more than a month ago
  2. RO CSVI
  3. # 30
Accepted Answer Pending Moderation
So it worked with the 90K lines?

there is some oddity there.
Once the copy fails, you will get a not readible file message because CSVI checks if the file exists and it doesn't. That can be explained, just not why the copy works one time and fails another.
  1. more than a month ago
  2. RO CSVI
  3. # 31
Accepted Answer Pending Moderation
I went back today and tried it without any mods, just import and preview, no issues, preview works fine.

I'm wondering, are you losing the path variable somewhere when an update to the source file is loaded? Maybe the file path? What are your error traps to generate that message minus the actions it performs? Is it a default for a case somewhere?

The fact that its not eating the file anymore, I think that is where the clue exists. The question would be, are you exceeding a memory limit on the server and the default is for the server to delete the file? I "think" my max memory is 128MB, does the import ever exceed that? How do you hand a memory overflow/limit server side? Maybe a static check of what value is needed vs. what the file may generate and compare?
  1. more than a month ago
  2. RO CSVI
  3. # 32
Accepted Answer Pending Moderation
Ok, I think we are chasing our tail here. I disable "cache cleaner" which I thought might have been the issue, it wasn't....every time I change the template or view it, and hit save and close, and try to reload the preview it fails and file is gone.....then it hit me.......its the TMP directory.....I'm wondering if the core cleans it up "whenever" it feels or whatever.....so I just said screw it....created a new directory called import and voila.....series of save and closes, navigation moves, previews....doesn't fail once....so I guess for me, instead of:

http://xxxxxxxxxx.com/tmp/myfile.csv

to

http://xxxxxxxxxx.com/import/myfile.csv

That seems to do the trick and a single source point upload for retention. Maybe this might help others, but my guess is the TMP isn't the best place to put your master file since it is subject to cleaning at any given point via a plugin/core process.....who knows....but this solution appears to fit my situation.

Thanks for all your time and effort Roland. I like version over all the rest thus far.

Chris
  1. more than a month ago
  2. RO CSVI
  3. # 33
Accepted Answer Pending Moderation
Hello Chris,

then it hit me.......its the TMP directory
That is quite the eureka moment ;)

but my guess is the TMP isn't the best place to put your master file since it is subject to cleaning at any given point via a plugin/core process
Agree, it is a temporary place after all.

Glad to hear this rabbit has been unmasked.
  1. more than a month ago
  2. RO CSVI
  3. # 34
  • Page :
  • 1
  • 2


There are no replies made for this post yet.
Be one of the first to reply to this post!