I have over 90K import to redo, at first all is well, but as I move to larger imports, it crashes after import preview.
I've tested 8700 import lines ok, went to 26000 lines and it causes it to crash, no error, no nothing....just white screen and stops.
Any ideas?
I've added the file here but is password protect as it now has private data I don't want people viewing.
Send me a private message and I'll give the password of the .csv file
Thanks Roland.
Chris
can't seem to upload it as file size is to big......and .rar won't go through...let me know how to get you the file.
Tried a list with 15000, that crashed too.....not sure what the limit is, but the files do fully upload and then it goes to the white screen and dies....no further action after that.....log shows nothing, just the start time, with an unknown event result.
Hello,
No need for the file, because "just white screen and stops" means you have an error but you have error reporting turned off. If you go to System -> Global configuration -> Server -> Error reporting and set this to Maximum or Developer and save the settings. Run the import again and you should see the error. Once we have the error we can investigate further or take action as needed.
Kind regards,
RolandD
=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
I tried that, there are no error messages, it just locks the browser up and stops.
Thoughts?
Anything under ~12K lines seems to load in just fine, anything over, this is the behavior I get.
I can give you access to try with two files I have.
Chris
I tried it again with different settings > dev > max > etc.
Nothing seems to allow me to see the issue.
Any further ideas? I need to get this sorted soon.
Thanks for your help.
Chris
Hello Chris,
it just locks the browser up and stops.
So this is not the white screen you were talking about before? If you enabled the Debug System option of Joomla, that should be turned off as it can lock up the browser pretty easily with such a large file.
One more idea, you say you are uploading the file, what if you FTP the file to the server and let CSVI read the file from the folder, do you get the same white screen?
Kind regards,
RolandD
=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
It is the same "lock up" with or without the debug mode on. Same white screen, but not total white screen, only the content area locks up. I can still "back" out of the page....just that the preview faults and stops leaving a white content area only. No debug info, no nothing.
I'll try to upload via ftp, I assume to the /root/tmp folder and look for it there? Never did the ftp method. Should be easy I suspect.
I am getting this via FTP which may point to a problem in the upload version too:
The specified file /homepages/redacted/redacted/redacted/test.aaron-beachproductions/tmp/failed.csv is not a readable file
It was passed this file over and started to try an open the index.html until I gave it an exact path and file name......this was the result.......so something about this file it doesn't like, but its the same script and output used for the 12K or less which is fully operational via uploading.
Hello,
I'll try to upload via ftp, I assume to the /root/tmp folder and look for it there? Never did the ftp method. Should be easy I suspect.
This can be any folder you like as long as it is readable by CSVI. This could be /homepages/../test.aaron-beachproductions/tmp/. That way it is in the same location as the site.
The specified file /homepages/redacted/redacted/redacted/test.aaron-beachproductions/tmp/failed.csv is not a readable file
This message is generated by CSVI after it check if the file is a file or folder. These are checks done by PHP, so it would suggest to me it is something with either access rights to the folder or file that are not correct. This can happen if the FTP user is different from the Server user.
Another thing I was thinking in regards to the upload issue (which stores the file as the Server user) is that the CSV file may be too large and hits the server limit for allowed uploading of files. That is something you could check as well. Usually this limit is set around 2MB by hosters. You can check with your hoster what the limit is for your site and perhaps it needs increasing.
Kind regards,
RolandD
=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
I checked, its 40MB and the file is only 15.
There is something really wacky going on here, check the image, you'll see what I mean and why I'm stumped
Hello,
This is not so wacky once I tell you why ;)
You put in the folder name and not the complete path to the file. So CSVI finds the first file, which is the index.html file in that folder. After the tmp put Failed.csv so the path ends with /tmp/Failed.csv. Save the template after that and give it a try.
Kind regards,
RolandD
=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
That is how I got the error I mentioned about unreadable file. Here it is as you suggested:
A good night sleep later and I may have a lucid moment. The error states failed.csv and your FTP screenshot shows Failed.csv. On linux servers these are not the same file. So in your template change the filename to Failed.csv or perhaps better rename the file on the server to failed.csv.
Give it a try then.
Kind regards,
RolandD
=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
Good point, thank linux.......LOL. But that wasn't it either, something else is going on when the import works, its destroying the file:
http://www.kittmaster.com/imagedump/csvi/ImportCSVI.htm
CHeck that out.....could be the issue at hand.
Thanks,
Chris
Hello Chris,
That is a good find but still has me confused. What happens is when you go to the preview, CSVI makes a copy of the original file, so it can do with it what it wants. It seems it is the copy process where it fails.
If you can send me a private message with Joomla and FTP access, I can check the issue further to see what happens. Are you using the FTP layer by any chance?
Kind regards,
RolandD
=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
Message and data sent.
Data received, could you please put the file in the tmp folder? I don't see it at the moment.
Kind regards,
RolandD
=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
Uploaded.....TY
Hello Chris,
Look at the attached image, without any changes to the system.
Which browser are you using? In my case I used FireFox, altough I fail to see how that would be of influence.
Kind regards,
RolandD
=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
Using Chrome.... :whistle:
Works fine in Chrome for me as well :evil:
Kind regards,
RolandD
=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
hmmm......let me give it a go again......see what happens
ummmmmm yeah.......seeing it work now here as well......the video I shot was what it was doing from day 1......strange......and now I see that the .csv file is not being deleted anymore either.......what gives there?
Now it automagically works.......and you didn't change a single thing?
I didn't change anything, my first step was to reproduce what your video shows. I didn't get there because it just worked. Perhaps a change on the server?
Kind regards,
RolandD
=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
Nothing on my end.....in the least.....I'm using 1and1.....they suck in updates......but would be really strange to have change during this problem set.....
Well, whatever, seems to be working now, if it happens again, I'll give a shout....
Just as an FYI, only the file size max upload limit would determine the import size I can put up there right? If 26K is 15MB, I suspect I could do a half and half to all 90k done within two files sets?
There are no hard limits in the code is I guess what I'm asking for max import line items?
There are no hard limits in the code is I guess what I'm asking for max import line items?
No hard limits. The maximum lines per batch can be configured in the template to work around the maximum execution time.
I suspect I could do a half and half to all 90k done within two files sets?
You can do that or FTP the full file and have CSVI read that one. In that case the file upload limit is not in play.
Kind regards,
RolandD
=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
Ok, tried that theory and updated....I merged all the data....and went to do the preview again:
test.aaron-beachproductions/tmp/Master-All.csv is not a readable file
It has since deleted the file again like the video.
ARGGGGGGGG
That is very odd as we both have seen it working before as well. Perhaps something on the server gets triggered? CPU usage for example?
I did release CSVI 6.2.0 just now, it wouldn't harm to at least update to that release and see if it makes any difference.
Kind regards,
RolandD
=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
6.2.0 Same thing, eats the file and says test.aaron-beachproductions/tmp/Master-All.csv is not a readable file
EDIT: It didn't eat the file, just error.
I went and did it again, see 90K lines....did it again......90K lines.....there is some oddity there.
So it worked with the 90K lines?
there is some oddity there.
Once the copy fails, you will get a not readible file message because CSVI checks if the file exists and it doesn't. That can be explained, just not why the copy works one time and fails another.
Kind regards,
RolandD
=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
I went back today and tried it without any mods, just import and preview, no issues, preview works fine.
I'm wondering, are you losing the path variable somewhere when an update to the source file is loaded? Maybe the file path? What are your error traps to generate that message minus the actions it performs? Is it a default for a case somewhere?
The fact that its not eating the file anymore, I think that is where the clue exists. The question would be, are you exceeding a memory limit on the server and the default is for the server to delete the file? I "think" my max memory is 128MB, does the import ever exceed that? How do you hand a memory overflow/limit server side? Maybe a static check of what value is needed vs. what the file may generate and compare?
Ok, I think we are chasing our tail here. I disable "cache cleaner" which I thought might have been the issue, it wasn't....every time I change the template or view it, and hit save and close, and try to reload the preview it fails and file is gone.....then it hit me.......its the TMP directory.....I'm wondering if the core cleans it up "whenever" it feels or whatever.....so I just said screw it....created a new directory called import and voila.....series of save and closes, navigation moves, previews....doesn't fail once....so I guess for me, instead of:
xxxxxxxxxx.com/tmp/myfile.csv
to
xxxxxxxxxx.com/import/myfile.csv
That seems to do the trick and a single source point upload for retention. Maybe this might help others, but my guess is the TMP isn't the best place to put your master file since it is subject to cleaning at any given point via a plugin/core process.....who knows....but this solution appears to fit my situation.
Thanks for all your time and effort Roland. I like version over all the rest thus far.
Chris
Hello Chris,
then it hit me.......its the TMP directory
That is quite the eureka moment ;)
but my guess is the TMP isn't the best place to put your master file since it is subject to cleaning at any given point via a plugin/core process
Agree, it is a temporary place after all.
Glad to hear this rabbit has been unmasked.
Kind regards,
RolandD
=========================
If you use our extensions, please post a rating and a review at the Joomla! Extension Directory
Log in to reply to this topic.