Dual monitors jump between screens

My computer both at home and work has 2 monitors. When I remote in to the office my screens at home jump between screen 1 and 2. There is a delay until I can use my mouse or enter data until it stops jumping. Any ideas?