Let me present an argument in favor of the ternary operator. tl;dr -- It is an abstraction on a conditional assignment pattern.

We start with your "readable" approach:

    var a;
    if (condition(x)) {
        a = 4;
    }
    else {
        a = 5;
    }

That has all the glory of line noise, repetition, and a declaration to make our language happy. This example is a little exaggerated. We can make our declaration more useful and our condition more of an exception:

  var a = 5;
  if (condition(x)) {
      a = 4;
  }

That is still two assignments or, rather, the "a =" pattern repeated. And that is just for a trivial single branch. How can we abstract that out? You have two options. For simple conditions, we can do this:

  var a = condition(x) ?
              4 :
              5 ;

It might be better to abstract it out like this:

  var a = value_for_a_with_environment(x);

Sure, you may be using a ternary operator or if...else underneath the hood in value_for_a_with_environment(), but give that function a relevant name, and someone can get the gist without having to know the details.

It could just be arrangement and syntax we despise. Here is another language's approach to the ternary operator:

  (setf a (if (condition x)
              4
              5))

Or with consideration given to the environment:

  (setf a (value-for-a-with-environment x))

Once I learned some of the value and abstractions of functional programming, I stopped feeling the ternary operator's use was ugly and realized it was more that I hated the line noise involved in using it. That colon looks a lot like a semicolon.

Also, it is limited. In nontrivial code, I have seen cascades of ternary operators.

  var a = cond1(x) ?
              4 :
              cond2(x) ?
                  5 :
                  cond3(x) ?
                      6 :
                      7 ;

or

  var a =   cond1(x) ? 4
          : cond2(x) ? 5
          : cond3(x) ? 6
          :            7 ;

Pick your favorite arrangement.

We really want a non-binary-looking structure like this:

  (setf a (cond (cond1 4)
                (cond2 8)
                (cond3 12)
                (t nil)))