First, I think there is a bit of confusion on how these parameters work so I will try to clear that up. But I believe your sentiment is valid. I will first discuss how these parameters work with each other and then discuss why.
How:
The memory-size for a particular namespace is the maximum amount of data that namespace can store in RAM. The high-water-memory-pct is a threshold that activates the namespace supervisor (nsup) thread’s eviction routine. The threshold is determined by the amount RAM you are using divided by the amount of memory configured for this namespace.
So to use your example, if you have 1TB of available RAM and configure you namespace to 600GB and high-water-memory-pct to 60 then eviction will start at 360GB. Though NSUP now evicting, depending on how NSUP is configured it may not evict fast enough so you may still reach the 90 stop-writes-pct or 540 GB. Stop-writes doesn’t actually mean that all writes stop, just the writes originating from a client. Stop-writes will continue to allow replica writes and migrations to pass through.
Why:
For the why I will refer you to this article explaining some of our choices in configuration defaults: