I have a custom Visual Studio workflow. There are hundreds of instances of it running in a site collection. I now need to update it. The workflow consists of some custom task forms and the workflow itself and I have WSP files built for all of these.
Should I use Install-SPSolution or Update-SPSolution so that the existing instances of the workflow keep running without being affected by the update?
Update-SPSolution is what you need, assuming you did not add any new features to the workflow. If you did, you need to use Install-SPSolution
Check out this link referencing upgrading workflows: https://msdn.microsoft.com/en-us/library/aa543659.aspx
In both, the DLLs and the files will be updated in the GAC the same way, but if the solution has changed the ID of a feature, changed scope of a feature, changed version of a feature, changed elements.xml, or has a new Property element to the Feature.xml file, then a complete retraction and reinstall will have to be performed--remove the workflow solution and remove it from the farm entirely before Installing.
Updating the work flow's wsp is little shady. You can use the Update-spsolution to upgrade it but what about the running instance of the Workflow? and it also depend what you change in the wsp.
Check this for more information.
