Resolved 8 hours to import a 4mb file with 17 fields and 12500 lines

Now I feel like you are dropping the ball on me. The data integrity is based on the products being imported by your tool. Am I seriously the only person that has 12k products importing into J2Store?

Hello,

It doesn't matter if you import 12000 articles or 1. If an entry needs to be created within the tree, this can lead to that 3-5 second delay depending on the server resources.

CSVI does do the integrity checks when you import data. If it didn't we wouldn't have to take care of the nested tree and you wouldn't see that delay.

I have no clue how you are deleting your products. If you just delete the Joomla articles, Joomla will clean up the tree. You gave the impression that you wanted to delete data directly from the database table.

I do not feel I am dropping the ball here. Contrary, I do think we have gone beyond regular support to help you figure out what is going on. We will assist further with your imports. However I do want to draw a line that we are not here for all your technical questions. We will help where possible but within reason. I have given my insights on how you could proceed.

Hi Roland,

I was deleting the products by dropping the data from tables directly using queries to do so. I was not deleting the assets table data.

You have gone above to support me on this issue. I do apologize for my comment. My frustration got the best of me. This is my first time using the J2Store component vs. using Virtuemart, which I have more experience with. Can you tell me if based on my data if you feel that J2Store vs. Virtuemart is not a good choice for this volume of data? I screened the topic with their support prior to making that decision. Now I am second guessing my decision. If I were to use VM, does it utilize the assets table the same way or is VM more contained within its own tables since it isn't using the Articles as the front end display?

I am concerned about the long term processes here. Of course, during the testing and quality assurance phases, the data layouts have changed based on what is available from the fields. Beyond separately adding the categories, I wanted my client to only have to do 1 import step, not multiple.

Hello,

I do apologize for my comment. My frustration got the best of me.

No worries, all good.

Can you tell me if based on my data if you feel that J2Store vs. Virtuemart is not a good choice for this volume of data?

The feeling I have is that you were just bitten by Murphy's Law, whatever could go wrong did go wrong in the worst possible order. It may prove worthwhile to just setup an empty site, with just Joomla + J2Store + CSVI and run your import and see how that goes. This will give you a good idea of what happens on a clean site.

Looking towards the future, the 1 million dollar question is, are you constantly inserting (not updating) 12,000 new products. If they are inserted in the middle of the tree, you will run into the delay. If they are added at the end, it won't be an issue. That is really looking into a glass ball. Updating records isn't an issue since we don't need to create the entry in the tree.

If I were to use VM, does it utilize the assets table the same way or is VM more contained within its own tables since it isn't using the Articles as the front end display?

VM doesn't use the asset table, this is just a by-product if you will from J2Store using Joomla articles. VM is only using it's own tables.

I wanted my client to only have to do 1 import step, not multiple.

By now, I am not fully clear anymore on what steps/imports had to be run. If there are multiple steps you could still make it easy for the client to run a cron job to run the imports needed based on whatever file the client has put on the server.

I understand the desire to do everyting in a single import but it is also impossible to enter product, category and manufacturer info into a single view in J2Store/VirtueMart. They are different data types. That is something to keep in mind. If the basics are all set most of the time you can do with a single import.

I guess the first decision to take is whether you want to continue with J2Store or switch to VM.

Hello,
I do apologize for my comment. My frustration got the best of me.

No worries, all good.

Thank you!

Can you tell me if based on my data if you feel that J2Store vs. Virtuemart is not a good choice for this volume of data?

The feeling I have is that you were just bitten by Murphy's Law, whatever could go wrong did go wrong in the worst possible order. It may prove worthwhile to just setup an empty site, with just Joomla + J2Store + CSVI and run your import and see how that goes. This will give you a good idea of what happens on a clean site.

OK

Looking towards the future, the 1 million dollar question is, are you constantly inserting (not updating) 12,000 new products. If they are inserted in the middle of the tree, you will run into the delay. If they are added at the end, it won't be an issue. That is really looking into a glass ball. Updating records isn't an issue since we don't need to create the entry in the tree.

The data is being managed on a different inventory server with no website interface. This is to give a website for their product catalog. At this point they have only set up a full export of products to be imported. They are working on modified dates, etc, but to keep it going that is how it currently is.

If I were to use VM, does it utilize the assets table the same way or is VM more contained within its own tables since it isn't using the Articles as the front end display?

VM doesn't use the asset table, this is just a by-product if you will from J2Store using Joomla articles. VM is only using it's own tables. That's what I thought.
I wanted my client to only have to do 1 import step, not multiple.

By now, I am not fully clear anymore on what steps/imports had to be run. If there are multiple steps you could still make it easy for the client to run a cron job to run the imports needed based on whatever file the client has put on the server. Yes, trying to make it as easy as possible.

I understand the desire to do everyting in a single import but it is also impossible to enter product, category and manufacturer info into a single view in J2Store/VirtueMart. They are different data types. That is something to keep in mind. If the basics are all set most of the time you can do with a single import.

These items aren't changing very much if at all, then a manual addition can be done. They would have to add it to the current inventory manually as well..

I guess the first decision to take is whether you want to continue with J2Store or switch to VM.

I'd hate to start all over. A lot of time has gone into this (at least from the troubleshooting. I guess it's worth looking at by now, but I thought I was close to being done.

Hello,

These items aren't changing very much if at all, then a manual addition can be done.

This way I don't think we have to worry too much about the delay, especially if the majority of the products are just updates. You could even specify in the template to skip any existing records, that way only new products would be added and existing ones are left alone.

I'd hate to start all over.

Given that the products change very little it might be an idea to run the big import and go from there. It might take some time but at least they are in and any subsequent import should be faster, especially if you skip existing products. If you do not skip them, then they would just be updates and not needing to insert anything into the asset table.

So to clarify, if I skip existing, but there is a change in the data, it would recognize that and import anyway? That is one of my issues. If a price changed or a description would it still import the modified info. Of course, we can unpublish the item if it is no longer available.

Hello,

So to clarify, if I skip existing, but there is a change in the data, it would recognize that and import anyway?

No, existing data will not be updated but only the new products will be imported.

Just wanted to give you a heads up. I started a VM test. I will report back to you after a few trials.

Ok, thanks.

Hi Roland,

Just wanted you to know I have successfully imported all the products 12491 in 11 minutes with image processing. Without it was about 5.

Huge difference. I wish I knew then what I know now .

Hello,

Thank you for the update. Of course hindsight is 20/20 but it has been a learning experience for sure. What I realized the other day is that such an issue can also occur for those using just the Joomla content import as it uses the same tree. This looks more like a core Joomla issue than anything else. Sure, adding 1 article by hand is not as noticeable as importing 12,000 items

Glad to see you on the path of recovery, have a good weekend.