Resolved Handling commas in field data

Hello,

Commas in field data is no problem. The CSV standard accounts for this by requiring text enclosures. So a line of data looks like this:
ABC,"This is a line, with a comma, or two",10.75

Yes, I must have some bad data then. The original exported file is " " enabled. I was able to get an cronjob import of 143k records, then it stopped.

I wanted to ensure that it wasn't a problem with " " as the text separator.

Hello,
You can check your import file for any errors using CSVI Analyser.Correct the errors and see if your import goes good after that.

that would be great but I have 2.5 million lines....... I don't think the analyzer can handle that. I am forced to manually test by chunks of data (which when I do that they are importing). Finding the needle in haystack is the issue here.

Hello,

The analyzer is not going to like 2.5 million lines

Can't you open the file in OpenOffice or LibreOffice and save it again as a CSV file, as explained in the Save a CSV file as UTF-8 tutorial. That should give you a valid CSV file.

Just thinking outside the box...

Yes, I have had some minor success with the data, but the whole file fails. When I do chunks of imports, they work fine. I have to find the bad data and it is a very large file.

I have reviewed the file with notepad ++ and with Tad (for visual columns), but reading each line is probably not the answer this way.

Hello,
Can you post a screenshot on the error you see when you run the import of whole file? Just want to check if it is not a timeout issue while importing such a huge file. You can also take a look at Manage Timeout document if that is the error you have.

I'm not getting any error on the screen, literally nothing. I was watching the phpmyadmin and cron job in the CP, No errors in the site log file either that are recent.

I will try again with changing the limits I have.

Hello,

If you get an invalid CSV file saving it with OpenOffice/LibreOffice/Excel then there is something seriously wrong. They save a valid CSV file. Is it not the server terminating the import due to load/duration or some other limit?

I'm trying to figure out the log file. I don't know now if it is the commas. I did an import with 60 records and it imported fine. I think it is the file size. Do you have any recommendations on a file that is 186mb, 2.5 million lines and what the settings should be?

Thanks again.

Hello,
For such large files we recommend to run the import using command line or setting it as a cron job. Import with cron or command line does not run in batches as it does for backend manual import. If you are running it manually then you can set limits to import by setting the number of records to be imported in each batch. Take a look at Manage Timeout document which explains on setting these import limits. Manual import will take a lot of time with such big file and it is possible to get into server timeout issues.

One other option is that if possible you can break the large file into smaller ones and put them in a folder. In CSVI import template settings set the folder path on Source tab. CSVI will then read all these files from folder and import them. See if this works for you.