It’s all about level of accuracy, and assurance equipment will remain accurate for months or years across a wide range of operating conditions. A VFR pilot flying in uncontrolled airspace has little need of an altimeter as they can see how high they are and if that altitude is enough to clear obstructions.
An IFR pilot has continuous, critical need of an extremely accurate altitude reference. Without an accurate altimeter they cannot be assured of clearing obstacles they cannot see because of weather obscuration, or fly at assigned altitudes that assure they won’t collide with other aircraft. Shooting an instrument approach in IMC conditions (you can't see the ground) requires tracking a radio beam glide slope, usually a 3* gradient, that becomes increasingly sensitive the closer you get to the airport. Every flight test level uses maintaining an assigned altitude tolerance as part of the test qualifications. As rating levels increase that tolerance decreases to where an Airline Transport Pilot should be able to manually fly and maintain an altitude of ~20’ or less of the assigned altitude. The cost of STC, PMA, and other test certifications can easily run into the millions of $$ before FAA acceptance is awarded. Aviation grade quality, reliability, and and accuracy has a price, one we don’t pay.
For our purposes, being limited to VFR and LOS conditions, the only altitude that concerns us is the maximum altitude we are permitted to fly at. Everything else is pretty much meaningless. We can always see the aircraft and clear obstructions using our eyes and through the camera lens.