Because that was originally the uniform reserved for doing manual labor or fighting in. The other one was formal occasions.
Why Are Military Uniforms Called Fatigues?