Resolved Google Shopping Export - couldn't connect to host

Hello Tim,
Glad to know you found a solution for your issue

Hi Roland

Just thought i would update you on the above.

Have set up the cron jobs, but unfortunately we have the same issue with unable to connect with this also - worth a try!!

I have found a work around, in fact its how we use to use sef urls on previous versions of csvi. Instead of using the product_url field on our exports we add the sef url to the intnotes field on vm when creating the products. Then on export we instead use this field instead with column header "link"

Looks to be working ok so far!

Kind regards

Tim

Hi Roland

Let me give the cron job a go as i wanted to look into how to set this up anyway.

Strange thing is i have csvi on other sites with same host and this is not an issue and functions properly. I didnt want to highlight this to them in case they blocked those too!

I'll let you know how cron job goes

Kind regards

Tim

Morning Tim,

Looks like the root cause has been found

Our servers prevent scripts making HTTP (port 80) connections back to the same server that they're running on, as policy.

. That is exactly what CSVI is doing, as you can see from the code sample I sent you. CSVI is calling a URL on your server, this goes through port 80.

Is there any work around we can do or i can present to our host?

The only work around I can think of is a cumbersome one. This involves running CSVI on a different host, set the domain name to the original domain name. You should be allowed to make a connection then. This fills the sef_cache table on the different host. This table would then need to be copied to the live site.

Did you try running the SEF URL update as a cron job? I wonder if that is seen as a loopback call. If not, that would be the easiest solution.

Morning Roland

Thanks for your reply, i have queried this back with our host and they have replied as below

"It does seem from your reply that this process may constitue a loopback connection on the server. Could you confirm if this is the case with your developer? Our servers prevent scripts making HTTP (port 80) connections back to the same server that they're running on, as policy.

This is done to prevent the possibility that mis-configured scripts could enter, essentially, an "infinite loop", making call-backs to themselves. Such a situation is extremely detrimental to server performance, and would negatively impact other users also on this shared server.

Unfortunately we are not able to lift this block on a per-site or per-script basis, so you will need to see if you are able to reconfigure the script (or contact the script authors for assistance doing this), or use an alternative script that doesn't require HTTP requests to its own server."

Is this how the sef caching works? Is there any work around we can do or i can present to our host?

Kind regards

Tim

Hello Tim,

It's hard to believe maybe but this is good news, now we have determined the root cause of the error you see. Since this is an easy way to reproduce the error, I am afraid it is back to your host to see why that request is blocked. This is really something outside of CSVI, so we have no control over it. Now that you have easy sample code to reproduce the issue, perhaps that can help your host as well to determine why the request is blocked.

Hi Roland

Thanks for your reply.

Have made that change and it shows pretty much same error message as attached?

Kind regards

Tim

Hello Tim,

There is nothing in particular on the server that will be blocking connections to the database.

It is not the connection to the database but to the website itself.

So digging a little deeper, the message "couldn't connect to host" comes from cURL. CSVI uses cURL to make an outbound connection. Perhaps this article can shed some light on this. Perhaps the server is blocking POST requests that miss certain parameters, that I can't see.

So I tried to send a few requests to your site and the answers came back as expected. If you want to try it out yourself, since you are coming from the same IP as the script here is the code:

$parseUrl = json_encode(

		array(

				'index.php?option=com_virtuemart&view=productdetails&' . 'virtuemart_product_id=10' . '&virtuemart_category_id=2' . '&Itemid=1',

		)

);

$http     = JHttpFactory::getHttp(null, array('curl', 'stream'));

$result   = $http->post('http://YOURDOMAIN/index.php?option=com_csvi&task=sefs.getsef&format=json', array('parseurl' => $parseUrl));

?>

	<pre><?php

echo __FILE__ . '::' . __LINE__ . ':: ';

echo 'result: ';

echo '<div style="font-size: 1.5em;">';

print_r($result);

echo '</div>';

?></pre><?php

Replace YOURDOMAIN with your own domain name. This code can be placed in the file administrator/components/com_csvi/csvi.php after line 34. When you open CSVI, you should see the response from your server. Would be interesting to see what comes back there.

Hi Roland

I have raised this with host and they replied as follows:

"There is nothing in particular on the server that will be blocking connections to the database. When you are trying to connect, are you using the external IP address of 46.32.241.98 or the internal one, which is 10.16.16.12? If you are trying to connect from the website itself, you should use the internal one but for all other purposes, it would be the external IP address"

Not sure if this helps?

Kind regards

Tim

Hi Roland

Yes the maintenance task produces same:

Error
couldn't connect to host

error message.

The site has 13831 products on, but even when i set the template to just one single product line with SEF urls on it produces same error.

Kind regards

Tim

Hello Tim,

The way SEF caching works is that it sends a POST request the following url http://example.com/index.php?option=com_csvi&task=sefs.getsef&format=json and the parseurl variable that is posted contains the URL or URLs to convert. During a regular export the parseurl contains a single URL but during a maintenance task it contains up to 500 URLs. The script resturns the generated SEF URLs as an array.

So you could check if some kind of limit is hit when posting to this URL.

If you don't have any SEF cache enabled, exporting 2500 URLs means 2500 requests to the front-end of your site. This could be seen as a DDOS attack and therefore not allowing you to send any more requests. That is where the maintenance task for SEF URLs comes in. With 500 URLs per request, the 2500 requests become 5 requests. This is significantly lower.

Did you try running the maintenance task to populate the SEF cache?

Hi Tharuna

Nothing in .htaccess that refers to csvi

Is there any other way we can test what is causing the csvi to be blocked?

Kind regards

Tim

Hello Tim,

but is there an external IP address i should check that spiders the pages accociated with CSVI?

There is no external IP address for generating SEF URLs. It is that CSVI is trying to access the front end and is been stopped. Can you also check your .htaccess file to see if CSVI has been blocked in there?

Hi Tharuna

I can see anything obvious in any of these security extensions. They block primarily by IP address, i am not sure how the SEF caching works, but is there an external IP address i should check that spiders the pages accociated with CSVI? If so i can add this to whitelist.

Export with SEF urls OFF works great with up to 2500 lines on, but idealy would like to use SEF urls.

Any other thoughts on what may be preventing SEF URL creation?

Kind regards

Tim

Hello Tim,
You can check if CSVI has been blocked in any of these firewall extensions and also in your .htaccess file. I think once you whitelist CSVI, you should be able to run the export.

Hi Roland/Tharuna

Yes using 7.1.0 now and it does seem to be the SEF URL's as it does work when turning this option off in the template. As this is the first export there will be no cache as yet., and running this refresh under maintenance generates same error couldnt connect to host error.

I wonder if somehow either admin tools or RS firewall has blacklisted the IP address generating the SEF urls? Wonder if there is an ip address i should check if blacklisted (as these SEF urls works fine on a couple of other sites i have sitting on different web spaces on same server using same set up)

Does that sound possible?

Kind regards

Tim

Hello Tim,

I believe Tharuna is pointing in the right direction and that it is related to the SEF URLs. This error message is not one that is created by Joomla or CSVI. Assuming you have CSVI 7.1.0 installed, you can also try to update the SEF URLs from the maintenance menu, and see if that works.

The export should then take the URLs from the SEF Cache rather than calling the server for every single URL and thus lowering the demand for resources.

Hello Tim,
The problem is only with Google export? What about any other export? Does it work?

I have a doubt with SEF URL generation in google export. The SEF URL is generated by calling the front end so probably that is reason you are seeing this error. Can you try setting Export SEF URLs to No on options tab in your template settings and see if export works?

... also cant seem to get any error message on server logs.

Kind regards

Tim

Hi Roland

No problem screen shot attached



yes mysqli driver

Happy to ask host provider if any reason why cant connect to database server, but would this not effect imports also?

Kind regards

Tim