Simple import got stalled #75798

Hi Tharuna,
thank you very much for your reply!!

It's not a cron import. Let me try to explain:

1. In my client computer, they publish some content in their custom python APP. This App uploads a csv file with this content. Then from this python app, they call the url to import the process.

I have asked my client for the response of the script, but he only gets the HTML response of the frontend import cronjob. So there is no "shutdown" signal from the client as they perform a CURL call as fas as I know.

We are in Joomla 3.10 and CSVI will be the replacement for our current import content system that will be in place once we update to Joomla 4.

I did not know there area a 8.7 version compatible with Joomla 3. Is it safe to use on Joomla 3?

Carlos

Hi Roland,
thank you very much for your replay.

When an import is stalled, is the file actually there?

Yes. It is there. But as you see in the logs attached, it just doesn't go through it.

you can also set the template to load the file from the server instead of retrieving it from a remote location. In case the filename is not always the same, you can just specify the folder and RO CSVI will import all files in that folder. Perhaps that is an option to look into. This way you remove the dependency on an external file download.

I tested this, but it made no difference. In any case, I will happily use this setting, so I will switch to this approach.

In any case, as I mentioned, it didn't help on my tests.

Anything I can do to get more information? I guess I could add some code somewhere to get more info in the logs...

Best!!

Carlos

Hi Roland,
attached you may find the log after changing the setting.

Best!!

• com_csvi.log.1.zip

Hi Roland,
I know!!! There are nothing else in server logs. This server does not provided a PHP error log, just the APACHE error log, but the call in the regular access log shows a 200 code, so my guess is that there is no PHP error around.

Also, if I call another PHP script calling the frontend CRON URL from my browser, the import works with no issues.

If I could call the PHP cronjob command from my PHP script, that could also be an option it might work, but not sure if that is a possibility.

Regards,

Carlos

Hi Roland,

What do you see when you run the CLI import? Can you run it directly from CLI yourself? How to run the CLI script is explained in the Setting up a cron job document.

Originally I configured it with a CLI import in a cronjob and we did not see issues with the import. But there were other issues related to the import that prevent us for using this approach. Basically we need the process to be started by this other remote server.

I know it is not ideal, but basically this approach simulates a remote post creation.

Is there a way to reliably reproduce this issue with the front-end cron?

Not from the browser. At the beginning we tried to trigger the import from the same PHP code that was called into our server with this code:

$ch = curl_init();
        $post_data = array();

        $url='https://CSVI-SITE';

        curl_setopt($ch, CURLOPT_URL, $url);
        curl_setopt($ch, CURLOPT_POST, TRUE);
        curl_setopt($ch, CURLOPT_POSTFIELDS, $post_data);
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
        $user_agent='Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0';
        curl_setopt($ch,CURLOPT_USERAGENT,$user_agent);
        $response=curl_exec($ch);
        curl_close($ch);

This was when our problems started and now we do a 2 steps approach triggering the frontend import from the external server just after the csv upload, with the same results.

Hi Roland,
thank you very much for your understanding and following up this thread.

I have noticed that the "POST" request sometimes works, so I'm not sure if that is the cause, but to be sure, I have changed to a GET request. Let's see if that fixes the issue.

Best!!

Carlos

Hi Roland,
after changing the command I got a different result, but not sure if it will be useful though...

Before this test I removed all the log entries and running processes from the site.

Here is what I see:

1. The date reflected in the log file (attachment com_csvi.log.2.zip) is incorrect. It says 2023-04-17. I have attached the screenshot of the log in the listing with the correct date and time.
2. The content reflected in the log file is actually one imported on 2023-04-17. That makes no sense because before my test I cleaned up all the logs and the csv file we read is overwritten several times every day.
3. The actual content of the file was not imported, but immediately after I noticed this test, I checked, and I performed an import using the frontend feature and the content was correctly imported. The date in the log is wrong in any case (attachment com_csvi.log.3.zip)
4. This has left my system with 3 running processes that I cannot relate to any of the log entries because the created tmp folders are not referred in any of the 3 log entries I have.

I hope this makes any bell sound. I'm actually disturbed by the log dates in any case and the old data trying to be imported...

• com_csvi.log.2.zip
• 
• com_csvi.log.3.zip

Hi Roland,
thank you very much for your reply, but I'm afraid I think you need to rethink your log system. But, please let me explain my latest thoughts and focus on having a successful import.

The last test I did was trying to replicate the process from my home server and getting the curl response from teh php script.

The test is simple. I just call a PHP script from my computer command line:

 wget - "https://mysite.es/scriptfolder/test.php"

Inside the test.php file the relevant code is this:

$url = "CSVI FRONTEND CRON URL";

curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HTTPGET, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

$user_agent='Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0';

curl_setopt($ch,CURLOPT_USERAGENT,$user_agent);

$response=curl_exec($ch);
curl_close($ch);

print 'response=('.$response.')';exit();

This test get me the HTML code for the frontend import and then I noticed this code in the HTML:

<form method="post" action="index.php?option=com_csvi&view=imports" id="adminForm" name="adminForm">
			<h3>Template Hamaika Articles Import</h3>
			<div id="import-div">
				<div class="span2">
					<span class="badge badge-info">Processed</span>
					<div id="processed"></div>
				</div>
				<div class="span2">
					<span class="badge badge-info">Last server response</span>
					<div class="uncontrolled-interval"><span></span></div>
				</div>
			</div>
			<input type="hidden" name="task" value="" />
			<input type="hidden" name="csvi_process_id" value="1171" />
		</form>

and this JavaScript:

function startTime() {
			jQuery(".uncontrolled-interval span").everyTime(1000, 'importcounter', function(i)
			{
				if (30 > 0 && i > 30)
				{
					jQuery(this).html('The last server response has not been received in a timely fashion. The server may still be busy or stopped the process. You can wait a little longer to see if the server will sent a response or cancel the process.');
				}
				else
				{
					jQuery(this).html(i);
				}
			});
		}

So I think the problem is that your frontend import requires JavaScript to work. Am I right?

Which is a bit confusing as the PHP CLI does not need JavaScript to work... Is there a way to make this work without JavaScript?

Thanks for your help.

Carlos

PS: I promise to come back with the log issues when we fix this because the timestamps and the system is not helpful to fix issues like this.

Great!! We are getting there!!

If you can extract the 1171 from the output you can then call the following URL https://YOURDOMAIN/administrator/components/com_csvi/rantai/rantai.php?task=import&runId=1171. That should run for as long as the number of records is set in the template settings on the Limit tab under Lines per import. By setting this number very high it will run till the import is finished or the server kills the import :) That will all depend on how large the import files are going to be.

That seems something we can do using DOM parser, and limits are not an issue because it's just one line per import.

Considering other options, yesterday I was playing around with the cron CLI and I managed to make it work with this code:

include('cron.php');

Problems is that cli needs to receive a couple of parameters (key and template id) and I tried setting them before the include as this:

$template_id = '1';
$key = 'MY_KEY';
include('cron.php');

But they didn't make it through JInput in Cron, so I hardcoded them in a your cron.php file and it worked, but it's not ideal... Do you happen to know a way of passing the parameters to CLI when using "include" instead of the proper CLI? That would be a cleaner solution from my point of view...

Nevermind, I have successfully used the approach you suggested and it seems to work so far.

I'll start another thread for the log issues I found.