Resolved Content transfer from J3 to J4

Hello Frits,

There is no standard recipe for migrating articles from J3 to J4 but the process is rather straightforward as it is no different going from a J3 to a J3 site. What I would suggest is to do the following using 8.0.0 beta 2.

1. Create an export template in RO CSVI on Joomla 3 for Joomla content export
   
2. Add the fields you require. This can also be a subset as you can always add more fields in subsequent imports. Follow the Joomla Content import for the basic fields
   
3. Run the export
   
4. Create an import template in RO CSVI on Joomla 4 for Joomla content import
   
5. Import the file exported in step 3
   
6. You should see the articles now in Joomla 4
   

You can add more fields as needed, as I do not know if you use tags/custom fields/associations. All these can be exported and imported as well.

Let us know if you have any specific questions regarding this process.

Hi Roland,

I updated to 8.0.0 beta 2., unfortunately I keep running into issues. On import I now get this error message: "Missing required fields id or alias or title,catid or category_path", and no article is imported.

However:
- the alias and title are both in my export from the J3 site
- I assigned default values for catid and category_path in my import template in the J4 site

I thought that it should work this way with the templates?

Note: I cannot copy any ids (articles, categories or users) from the J3 site as they all have (or must have) different ids on the J4 site because we already created content (and imported from Blogger) on this site.

I noticed that in a preview of the exported J3 csv most of the introtext and fulltext content is not displayed. When viewing in a code editor on my local computer I see lots of line feeds inside these fields and double quotes in the text are not escaped. I manually edited these aspects in the csv for just one article and tried to import it.

However, I still got the same error message and the article is not imported.


Hope you can help me to solve this mystery,

Frits

Hello Frits,
Please post the import file you used along with import debug log to check. To get the debug log check How to collect debug information? document.

Hi Tharuna,

I have sent you the import file and the debug report by email.


Kind regards,

Frits

Hello Frits,
Yes got the email. Thank you for the files.

However:
- the alias and title are both in my export from the J3 site
- I assigned default values for catid and category_path in my import template in the J4 site

I thought that it should work this way with the templates?

You have added default values for catid and category_path in import template but the template still uses headers from import file so there is no catid or category_path in import. To set the template fields as headers, in your import template settings on File tab set Use file for configuration as No, save the template and run the import.

Let me know on how the import goes after that.

Thanks for your reply. Before creating this topic I tried both settings of the "Use file for configuration" parameter, i realize now that I did not mention that before.

I just switched it to No and I ran the template again. The error message does not appear again, however no articles are created either.

I'll send you the log report of this latest run by email.


Kind regards,

Frits

Op 26 okt. 2021, om 10:55 heeft RolandD Cyber Produksi <support@rolandd.com> het volgende geschreven:

Hello,
There seems to be some field mismatch, from the debug log i see that alias is mapped against image value. See the query from the debug log. To check more on this, please post the import template XML to check. To get the template XML file, in RO CSVI maintenance menu select RO CSVI on first option, backup templates on second option, select your import template and download the XML file. Post the XML file.

SELECT `id` FROM `#__content` WHERE `alias` = '{\"image_intro\":\"\",\"float_intro\":\"\",\"image_intro_alt\":\"\",\"image_intro_caption\":\"\",\"image_fulltext\":\"\",\"float_fulltext\":\"\",\"image_fulltext_alt\":\"\",\"image_fulltext_caption\":\"\"}' AND `catid` = 2

Hi,

I've sent you the xml and also a screenshot of the settings for the template fields as I made them in the CSVI backend interface.


Kind regards,

Frits

Hello Frits,
Thank you for the files. I found two issues with the files.

1. Your order of template field is wrong and so the values from import file are messed up. I have attached on how the order should be with a screenshot. Please rearrange the template fields. You can check the values on Import preview page to confirm if they are in right order.
2. Your import file is not UTF-8 encoded. There are some unreadable characters when checking the import file with RO CSVI Analyser. This should be something with the programme you are using in opening the export file from Joomla 3. Try OpenOffice/LibreOffice to open the export file in case you are editing it. Run the import with UTF-8 encoded file and check.

Hi Tharuna,

Thanks for your reply.

1. Order of template fields

How I can avoid this in the future:

• Does the order of the fields have to correspond to the order as in the Joomla database table #__content? Or is ther another rule that I can follow?
  
• Do I need to keep this order in the import template, the export template, or both?
  

2. UTF-8 encoding

The csv file I sent you comes straight from the CSVI export in the J3 site. It was not edited or saved using a (desktop) application.

Kind regards, Frits

Hello,

How I can avoid this in the future:

Does the order of the fields have to correspond to the order as in the Joomla database table #__content? Or is ther another rule that I can follow?
Do I need to keep this order in the import template, the export template, or both?

The order of the fields should follow the order of field values in import file. RO CSVI cannot set the values from import file automatically. It will follow the template fields and read the columns from import file.

UTF-8 encoding

The csv file I sent you comes straight from the CSVI export in the J3 site. It was not edited or saved using a (desktop) application.

It is possible that your database has those unreadable characters. You need to correct those characters before importing into Joomla 4 installation.

Hi Tharuna,

1. Order of the fields

Thanks for your explanation. While working on this I noticed another thing that I did wrong with the fields: I should have the columns in the csv that I intend to fill with a fixed value. I thought that I should leave those out because their values will not be used.


2. Character set

The site is an old one; although it is J3 now, is has a history that might even go back to J1.5. The character set of the database is utf8mb4_unicode_ci, all special characters (e.g.: é) are displayed as they should in the site, both in backend and in frontend.

Now that the issue with the fields is solved, I made a new test with a MySql export instead of CSVI. With this export I was able to import the articles successfully. Also, when previewing the MySql export in a code editor or in LibreOffice the characters are looking ok, which is not the case with the CSVI export. I'm have no idea what makes the CSVI export contain the wrong characters.

Howver, for me the main thing is that I now know how to make it work. And I'm happy to have learned this because in the near future there will be more J3 sites that I'll have to migrate to J4.


Thanks for your help,


Frits

Hello Frits,

1. Order of the fields

Thanks for your explanation. While working on this I noticed another thing that I did wrong with the fields: I should have the columns in the csv that I intend to fill with a fixed value. I thought that I should leave those out because their values will not be used.

I am not sure what you mean by "fill with fixed values" but the columns in CSV file or the fields you have added to template fields are the ones which are updated on import. The rest of the fields stays the same.

The site is an old one; although it is J3 now, is has a history that might even go back to J1.5. The character set of the database is utf8mb4_unicode_ci, all special characters (e.g.: é) are displayed as they should in the site, both in backend and in frontend.

That explains your database has no unreadable characters. Then it must be software you are using for the export file which does not encode file as UTF8 by default.

Now that the issue with the fields is solved, I made a new test with a MySql export instead of CSVI. With this export I was able to import the articles successfully. Also, when previewing the MySql export in a code editor or in LibreOffice the characters are looking ok, which is not the case with the CSVI export. I'm have no idea what makes the CSVI export contain the wrong characters.

That is really strange, even i have no idea on what is causing these wrong characters. RO CSVI does nothing with characters. It just exports what is in database to the export file. Did you directly open the RO CSVI export file in code editor or in LibreOffice? You still see those unreadable characters?

1. When I said "fixed value" I meant to say: "Default value"

2. I tried opening the csv file as exported by CSVI in PhpStorm now (before that I used a very simple code editor for the Mac that is not much more than a text editor).
With PhpStorm I get a message that says: The file was loaded in a wrong encoding: 'UTF-8', and it suggests reloading it as ISO-8859-1.
When I do that, the text looks ok.

In the attachment you see message and also you can see a wrong character in the text. After reloading as ISO, this character is rendered correctly as: è


Frits

Edit: if I export the same content directly from MySql, the export file is rendered correctly by PhpStorm, without any messages and displaying the charater è correctly. So I suppose there must be some difference between the MySql export and CSVI.

Hello Frits,

1. When I said "fixed value" I meant to say: "Default value"

Can you tell me which fields you had to set Default value? Can you please explain more on this?

2. I tried opening the csv file as exported by CSVI in PhpStorm now (before that I used a very simple code editor for the Mac that is not much more than a text editor).
With PhpStorm I get a message that says: The file was loaded in a wrong encoding: 'UTF-8', and it suggests reloading it as ISO-8859-1.
When I do that, the text looks ok.

In the attachment you see message and also you can see a wrong character in the text. After reloading as ISO, this character is rendered correctly as: è

UTF-8 is not wrong encoding but it is the default encoding type used in RO CSVI. We recommend users to use UTF8 encoded files for import as that is the standard used these days to support accent characters. Instead of opening the export file in code editors, try opening it with LibreOffice or OpenOffice.

So I suppose there must be some difference between the MySql export and CSVI.

The only difference i can see with your MySQL export and RO CSVI export is the charset of the file.