Yes you can update the custom fields for existing articles. You can either provide article id or title/alias and catid/category_path fields in your import. With these fields RO CSVI will check if there is any matching article and update the custom fields else a new article will be created with custom fields.
Can you update based on the article title (and category path)? The documentation states you need an alias to update.
I have a site where I import content from an external database to articles. But I don't have aliases (or article id's) in my import file.