The objects swap velocities only if their masses are equal. Why? Because if their masses are equal, m1v1 + m2v2 (initial) = m1v1 + m2v2 (final) reduces to v1 + v2 (initial) = v1 + v2 (final). The kinetic energy equation also reduces to v1^2 + v2^2 (initial) = v1^2 + v2^2 (final). This also assumes that the masses of both objects remain unchanged due to the collision. This works in a case where one of the velocities is zero. Remember that all collisions conserve momentum, and in elastic collisions, kinetic energy is conserved too.
Consider a case of two balls of equal mass colliding into each other head on at a velocity of v0. An elastic collision would result in them rebounding off with the same velocity v0 (notice how the velocities were swapped). A perfectly inelastic collision would result in them sticking together and stopping. Anything in between is an inelastic collision.
