Working with the pivote table data was not clear to me initially - especially deleting from the pivote table. I thought Laravel would handle this since it already handles the inserting and updating of the data. But it turns out this needs to be handled at the database level.
When setting up your pivot table, you need to set the referential actions to cascade on delete and update. This will force any update to the parent table to cascade down the data in the pivote table.
Below is a simplified example using a blog with posts and tags.