You do get improvements very quickly when you first start an exercise program - a lot of it is your muscles learning to work together and coordinate better. The other part of improvement is actual muscle development, and you will often notice that most after a rest day. When you exercise you actually damage and break down your muscles a bit, and the body responds by rebuilding them better than before. So the rest is vital periodically to make sure your body has the capacity to rebuild.