Returns to a factor refers to the effects on output of changes in one input with all other inputs are hold constant. On the other hand, returns to scale refers to the effect on total output of charges in some constant rate in all the inputs simultaneously.