Resolved image import

Hello,

and store it locally in my Joomla images folder, ideally in a dedicated subfolder.

That won't work because K2 uses a fixed folder where the images are installed. This folder is /media/k2/items/src/.

This is also the folder where CSVI expects the image to be as it gets renamed to the md5 hashed filename that K2 uses.

The fieldname for the image field is image. So you can add that in addition to the other fields you use for the import. At a minimum it is the alias and category_path fields.

CSVI cannot retrieve remote images for K2.

Are you using a dedicated subfolder now with K2? If so, how do you make that work? As far as I know, we only have the fixed folder.

Actually I don't mind where it stores it as far as it is accessible by K2.



This is my K2 media window. Ideally I'd like to specify somewhere a root url to download the images from and store it in the game folder of the K2 media space.

CSVI cannot retrieve remote images for K2.

That is real bad news as I purchased CSVI for its capability to import images. Is that feature only available for virtuemart ?

If so, what is the meaning of the image import tab ?
It looks generic whatever the import template we choose.
If it's only for VM do you considere adding the feature to K2, Joomla or as a separate import process.

In my case I absolutely need to get access to the image. I see a fe options. Which one would you recommend ?

1. I'm wrong and you have an image import processor for Joomla/K2 ..
2. It's only VM but you could have a beta version of the import processor for K2 to share.
3. I should consider using VM. Seriously it's possible although not my preferred path.
4. I should consider using extra field in K2 and use the remote image in my views
5. You have a better idea ..

please advice

This is my K2 media window. Ideally I'd like to specify somewhere a root url to download the images from and store it in the game folder of the K2 media space.

Is it me or this media manager is really slow? Anyway, it won't work store the images in the game folder for example, when you select an image from the game folder and store the item, K2 hashes the filename and places it in the media/k2/items/src folder.

That is real bad news as I purchased CSVI for its capability to import images. Is that feature only available for virtuemart ?

CSVI does import images, for K2 it doesn't do remote images.

If so, what is the meaning of the image import tab ?

For processing local images on the server that were put there via FTP for example.

If it's only for VM do you considere adding the feature to K2, Joomla or as a separate import process.

Not only VM but also EZ Realty.

In my case I absolutely need to get access to the image. I see a fe options. Which one would you recommend ?

Option 1 is true, since there is an image import for K2 just not remote images as you would like. Option 3,4,5 are not an option in my opinion. We stick with option 2, I will build the support for remote images and post a patch file, so you can have remote images import.

Sounds good?

Hi Roland,

Is it me or this media manager is really slow? Anyway, it won't work store the images in the game folder for example, when you select an image from the game folder and store the item, K2 hashes the filename and places it in the media/k2/items/src folder.

Now I get your point (finally). The thing is that I will need to import images for the product description but also additional media files for slideshows or video ... some will use the K2 storage, other will use component dependent storage. I will need to import remote data and have the import processor drop it in the right place and adjust the fields accordingly.

But one step at a time, importing the main image is my first target.

For processing local images on the server that were put there via FTP for example.

What is the process there ? If we consider that I have my media stored on a temp folder somewhere on the server, where do I indicate the temp source folder for the images ?
What should contain the image field to refer to the right image ?

Sounds good?

If you can have the remote import processor for K2, that's just great.

When could you have a version available ? I'm ready to help for testing or debugging.
I don't want to put any kind of pressure, but I'm on a project which needs this import stuff.

many thanks,

Hello Chris,

But one step at a time, importing the main image is my first target.

I am all for such approach

What is the process there ? If we consider that I have my media stored on a temp folder somewhere on the server, where do I indicate the temp source folder for the images ?
What should contain the image field to refer to the right image ?

You upload the images with their real name to the /media/k2/items/src/ folder. In your import file you only put the name of the file, for example myimage.jpg. During the import CSVI will update the images accordingly for K2.

If you can have the remote import processor for K2, that's just great.

I will implement it and post the patch when it's done.

When could you have a version available ?

This should be ready in the next couple days.

OK I got it, thx.
I assume that the image import processor for K2 that you'll provide will automate the first step, i.e download the image from the provided URL in /media/k2/items/src/ and the file name out of the url to save the item in K2.

Can it be used for categories as well ?

What about other media file import (galerie, video, sound file, ..)
As media mapping is very component specific, do you see a way to provide a more generic media import processor. I would see a form where we indicate a CSV file with remote url / component field mapping (ie http://www.myremoteurl/mydata/myfile.ext -> K2/item/images or
http://www.myremoteurl/mydata/mysoundfile.ext -> Joomla/media/music)

Ideally it would also allow us to decide if the data must be imported or stay remote if the component allows it. For instance a Youtube video vs a local K2 item image.

I assume that the image import processor for K2 that you'll provide will automate the first step, i.e download the image from the provided URL in /media/k2/items/src/ and the file name out of the url to save the item in K2.

Correct.

Can it be used for categories as well ?

Once I complete that work, yes.

What about other media file import (galerie, video, sound file, ..)

I have not looked into that yet as there has not been any requests so far.

As media mapping is very component specific, do you see a way to provide a more generic media import processor.

This is decided upon per component. Each component handles media in it's own way. The VirtueMart implementation has no fixed path so you can use what you like but only because VirtueMart allows for it. K2 is more rigid and expects the media in one folder and a hashed filename.

Ideally it would also allow us to decide if the data must be imported or stay remote if the component allows it. For instance a Youtube video vs a local K2 item image.

I don't think you get to decide that if the K2 team has decided it for you

I don't think you get to decide that if the K2 team has decided it for you

That's right, but K2 articles can contain data coming from other components or plugins, like galleries or video players and these ones can have other requirements for the storage, but you are right, it's not an issue at the moment.

When I'll have the image import for K2 work, this will be already a big step.

The patch has been posted in this thread.

I tried the patch with a K2 item import.
I have items imported but the fields are not repecting the mapping I made.

I get the short description in the category although I verified my mapping and everything looks OK.



here is the debug log :
com_csvi.log.1414573884.zip


Regarding image import where should I indicated the root url for the image. I'm expecting some configuration in the image tab to indicate something like http://mymasterserver.com/images

If in my CSV file I have an image named myimage.jpg, I'm expecting the import processor to concatenate a root url with the image name and process the image.
In my sample it would download the image http://mymasterserver.com/images/myimage.jpg, do the magic to drop it in K2 media directory and register myimage.jpg in the item image field.

I can't have the image data in the CSV with an absolute path.

thank you

I get the short description in the category although I verified my mapping and everything looks OK.

Post your CSV file as well so I can see where the error is. There must be a field order misconfiguration. The debug log shows the very first field in your import file is the category path, which it most likely isn't.

Regarding image import where should I indicated the root url for the image.

You should have that in the import file.

I can't have the image data in the CSV with an absolute path.

Why is that?

I'm expecting the import processor to concatenate a root url with the image name and process the image.

The import processor reads the data from your file, that is why it expects the data in the file.

What you can do is use a replacement rule to prepend the server name. See the Replacements tutorial. It has an example of how to prepend a value.

Post your CSV file as well so I can see where the error is. There must be a field order misconfiguration. The debug log shows the very first field in your import file is the category path, which it most likely isn't.

I think I forgot one step when I generated the map file. I'm checking now.
Is it mandatory to indicate 'Skip' on the field you don't need.
Can you have a fast way to set multiple field as skipped ones. It's painful to set them one by one.

Regarding image import where should I indicated the root url for the image.You should have that in the import file ... Why is that?

because The file is coming from a third party server. I don't master the format of the data.

What you can do is use a replacement rule to prepend the server name

That is a nice trick.

If I have multiple step to conform a csv file and proceed with the import, do you have some kind of chaining mechanism to automate the process. I will have ultimately to run it several times a week to add/remove/update the content on my sever.

I will use the cron task for that but I'm not there yet.

I think I forgot one step when I generated the map file. I'm checking now.

Definitely looks like that because the first field in the attached file is short description and not category path

Is it mandatory to indicate 'Skip' on the field you don't need.

Yes it is mandatory, otherwise CSVI will process the field.

Can you have a fast way to set multiple field as skipped ones. It's painful to set them one by one.

None that I known of other than pressing the letter S which will go straight to the first entry starting with the letter S. Press it again it goes to the second, etc.

If I have multiple step to conform a csv file and proceed with the import, do you have some kind of chaining mechanism to automate the process.

Not sure what you mean here however CSVI only does a single import run and no other tasks around it.

I will use the cron task for that but I'm not there yet.

A bash script is the way to go to have some management on import routines and related tasks.

ok, I'm doing some (little progress) now.


I create the replacement rule and another field mapping template for exporting my original CSV into a conformed version. The idea is to add the root URL to each image.
In my mapping file I added all fields I want to export in the new file.
In the field options of the template I added the replacement rule to the image field.

However, I'm not exactly getting the result I'm expecting.

1. I don't see how to ignore some field in the export (no skip in the field combo for K2 export).
2. I don't know how to apply the generated export template to another source file. Currently the export already knows the source file (from the mapping I guess) and don't allow me to select another file in the template tab like for import.

Regarding the image processing, what I want is to get the exact same file with the image field modified with replacement rule.As a result, the exported file has the root url happened before the image name as expected, but the image name is replaced with the K2 scrambled name instead of the original name.

ie :

http://www.mydefaultserver.com/images/products/2fa67f482133f1c934235b73c2a03954.jpg
instead of
http://www.mydefaultserver.com/images/products/myimage.jpg

I don't see how to ignore some field in the export (no skip in the field combo for K2 export).

You can't. If you don't want a field exported, just don't add it to the list.

I don't know how to apply the generated export template to another source file.

You will have to explain this further, as I don't understand what you are trying to do.

Regarding the image processing, what I want is to get the exact same file with the image field modified with replacement rule.

You can't either because the original name is not stored anywhere. K2 filenames are a hash of the word Image and the ID of the K2 item. K2 generates this on the fly to find the images on the server. There is no database storage of the main images.

but the image name is replaced with the K2 scrambled name instead of the original name.

That is because the original name is not stored anywhere.

If you don't want a field exported, just don't add it to the list.

In the field mapping, I can't add or remove fields. They are automatically generated from the source file. When importing data I can turn off the import of the fields I can't map to K2 by setting the parameter to 'skip'. In export I don't have such an option.

You will have to explain this further, as I don't understand what you are trying to do.

OK I will try to explain in detail what I did and what I understood :

My goal is to import a CSV file which can be refreshed quite often to K2.
The file contain a list of column with the various fields I want to import to K2 items.

1. I need to map the column names to K2 fields. That works with the map tab function.
2. I need to automatically create the missing categories. That works with the new delimiter option you provided with the patch
3. I need to import an image into the main item image. That fails because my CSV file contains only the name of the image and not the full URL to use to download the image. I would have liked an option in the image tab to specify the root URL for the images but it's not there and you suggested to use the replacement feature to add the root url to the image name.

That's what I'm trying to do and here is how I understood I should proceed :

For importing a CSV file :

1. I have a CSV file with multiple fields that I want to import to K2.
2. The field mapping tab allows me to create a map template.
3. To do so I create a K2/import mapping template, choose the CSV file to list all the available column/fields and associate some to K2 fields, skip the others.
4. From that mapping template, I generate an import template and name it, say K2 import
5. In the template tab, I load my K2 import mapping template, adjust a few parameters and save it back.
6. I the choose my CSV file to import and press proceed.
7. I double check in K2. This process works. but the images are not loaded yet.

From there, I need to use the new image import feature for K2.
My CSV contains the image name but not the url of the original server, from which I need to get the image to store in K2. You suggested to use the replacement option to replace the name of the image, say myimage.jpg with the full url, say http://myserver.com/images/myimage.jpg
For that I made the following :

1. From the replacement tab I create a new replacement template and edit it.
2. I set the parameters to add the root url before the image name as described in the doc. save the replacement template.
3. I understand that this replacement template is used together with an export process to create a new CSV from the K2 content
4. the generated CSV shall then be used to be download and store the images in K2 but ..

The generated CSV does not contain the original name anymore but the name exported from K2 which is already scambled.

In this scenario, how can I achieve my import task ?

You can't either because the original name is not stored anywhere. K2 filenames are a hash of the word Image and the ID of the K2 item. K2 generates this on the fly to find the images on the server. There is no database storage of the main images.

OK but then how to use the replacement option to alter the CSV file before it's imported.
My process seems not accurate and I don't see what step to follow.

In the field mapping, I can't add or remove fields.

Correct, the field mapper is only a tool to help you create a template from a file you want to import. In the template itself (Process page) you manage the fields after you created the template. There you can add/remove fields as you like on the Fields tab.

The 7 steps you outlined for running your import is correct.

I understand that this replacement template is used together with an export process to create a new CSV from the K2 content

Let's stop here, this is where it goes wrong. Replacement rules can be used by both the import and the export. In your case we are talking about the import. So now that you have the replacement rule, you go to the template and on the Fields tab, click on the image field to open the settings of this field. There you see the replacement rules also, select here the rule you created and click on Save. Now the replacement rule is used for this field. That should create the full image name in your case.

So you don't need to do any export. Hope this explains it better.

Let's stop here, this is where it goes wrong. Replacement rules can be used by both the import and the export. In your case we are talking about the import. So now that you have the replacement rule, you go to the template and on the Fields tab, click on the image field to open the settings of this field. There you see the replacement rules also, select here the rule you created and click on Save. Now the replacement rule is used for this field. That should create the full image name in your case.

Got it, now I have an error during import saying :

Cancel the import and check if all settings are correct and try the import again. Status error: 200 Status message: OK Warning: fopen("http://www.myserver.com/images/): failed to open stream: HTTP request failed! HTTP/1.1 403 Forbidden in /.../librairies/joomla/filesystem/file.php on line 327

It looks like the import is trying to open the root url and not the full url with the image name.
If with a browser I go to http://myserver.com/images/ I get the 'Forbidden' response.
If I go to http://myserver.com/images/myimage.jpg the image is displayed.

My replacement rules for the image field are:

Find : /^/
Replace with : http://www.myserver.com/images/
Multiple values : No
Method : Regular Expression

Do you have any rows where the image field is empty?

In any case, this is an issue with image processing, any kind of notice/warning/error will stop the import. What you can try is to go to the Global configuration and on the Server tab set Error repoting to None. So that all these messages are hidden and thus won't stop the import.

I just check in my sample file (only 25 rows to import).
There is an image file in each row.

So I guess the issue is elsewhere.

I'm having an issue with the field mapping.
I turned off the error reporting and now the 25 rows are imported but

The image name is used for category_name
the item name is assigned a skipped field (producer name)
...

see here




I double checked my mapping and I see no error. Here is the mapping I'm using :