When changing channels, only workers who are not currently assigned to a task will move with you. To ensure that all workers move with you, open your Worker UI, signified by a pickaxe in the upper left corner of your screen, below your level. Select “End Task” for every worker. This will keep them from repeating their current task, however, it will not cancel a task in progress.
Once all of your workers are idle, you can change channels and your workers will accompany you. If you change channels and your workers are in the middle of an assigned task, you will need to return to your previous channel, cancel all of your workers’ tasks, and wait for them to become idle, to move them to your new channel.
Please note that if you change channels with your workers set to repeat tasks, they will complete the task they are on, and stop working, without completing the full cycle of assigned tasks.