I can write a highly optimized piece of code that runs at 1ms or I can run an extremely readable piece of code that runs at 2 ms. In most cases isn't the correct choice readability > optimization? Prioritizing optimization over readability just means that less and less people can actually maintain my code.
I try to keep it readable for myself while optimizing it as much as possible. I rather add documentation if it becomes unreadable for others.