Resolved Large import causes white screen during preview

Using Chrome.... :whistle:

Works fine in Chrome for me as well :evil:

hmmm......let me give it a go again......see what happens

ummmmmm yeah.......seeing it work now here as well......the video I shot was what it was doing from day 1......strange......and now I see that the .csv file is not being deleted anymore either.......what gives there?

Now it automagically works.......and you didn't change a single thing?

I didn't change anything, my first step was to reproduce what your video shows. I didn't get there because it just worked. Perhaps a change on the server?

Nothing on my end.....in the least.....I'm using 1and1.....they suck in updates......but would be really strange to have change during this problem set.....

Well, whatever, seems to be working now, if it happens again, I'll give a shout....

Just as an FYI, only the file size max upload limit would determine the import size I can put up there right? If 26K is 15MB, I suspect I could do a half and half to all 90k done within two files sets?

There are no hard limits in the code is I guess what I'm asking for max import line items?

There are no hard limits in the code is I guess what I'm asking for max import line items?

No hard limits. The maximum lines per batch can be configured in the template to work around the maximum execution time.

I suspect I could do a half and half to all 90k done within two files sets?

You can do that or FTP the full file and have CSVI read that one. In that case the file upload limit is not in play.

Ok, tried that theory and updated....I merged all the data....and went to do the preview again:

test.aaron-beachproductions/tmp/Master-All.csv is not a readable file

It has since deleted the file again like the video.

ARGGGGGGGG

That is very odd as we both have seen it working before as well. Perhaps something on the server gets triggered? CPU usage for example?

I did release CSVI 6.2.0 just now, it wouldn't harm to at least update to that release and see if it makes any difference.

6.2.0 Same thing, eats the file and says test.aaron-beachproductions/tmp/Master-All.csv is not a readable file

EDIT: It didn't eat the file, just error.

I went and did it again, see 90K lines....did it again......90K lines.....there is some oddity there.

So it worked with the 90K lines?

there is some oddity there.

Once the copy fails, you will get a not readible file message because CSVI checks if the file exists and it doesn't. That can be explained, just not why the copy works one time and fails another.

I went back today and tried it without any mods, just import and preview, no issues, preview works fine.

I'm wondering, are you losing the path variable somewhere when an update to the source file is loaded? Maybe the file path? What are your error traps to generate that message minus the actions it performs? Is it a default for a case somewhere?

The fact that its not eating the file anymore, I think that is where the clue exists. The question would be, are you exceeding a memory limit on the server and the default is for the server to delete the file? I "think" my max memory is 128MB, does the import ever exceed that? How do you hand a memory overflow/limit server side? Maybe a static check of what value is needed vs. what the file may generate and compare?

Ok, I think we are chasing our tail here. I disable "cache cleaner" which I thought might have been the issue, it wasn't....every time I change the template or view it, and hit save and close, and try to reload the preview it fails and file is gone.....then it hit me.......its the TMP directory.....I'm wondering if the core cleans it up "whenever" it feels or whatever.....so I just said screw it....created a new directory called import and voila.....series of save and closes, navigation moves, previews....doesn't fail once....so I guess for me, instead of:

http://xxxxxxxxxx.com/tmp/myfile.csv

to

http://xxxxxxxxxx.com/import/myfile.csv

That seems to do the trick and a single source point upload for retention. Maybe this might help others, but my guess is the TMP isn't the best place to put your master file since it is subject to cleaning at any given point via a plugin/core process.....who knows....but this solution appears to fit my situation.

Thanks for all your time and effort Roland. I like version over all the rest thus far.

Chris

Hello Chris,

then it hit me.......its the TMP directory

That is quite the eureka moment

but my guess is the TMP isn't the best place to put your master file since it is subject to cleaning at any given point via a plugin/core process

Agree, it is a temporary place after all.

Glad to hear this rabbit has been unmasked.