Anton Geraschenko discusses the origin of the name in this post from November 11, 2009:
There are lots of reasons I think "MathOverflow" is an excellent name. Of course, I'm somewhat biased. It pays tribute to Stack Overflow, which I think is important because in addition to running the software developed for SO, MO runs basically the same philosophy as SO. If you do know about Stack Overflow, "MathOverflow" is the canonical name. When I had put up a beta
sign-up sheet for MO, but wasn't advertising at all, a lot of the
sign-ups came from an SO comment thread where somebody had
independently invented the name "mathoverflow" and somebody pointed
out that such a domain was actually registered and such a site was
actually going to exist.
While indeed (as Dmitri Pavlov points out) the first 2009 design of the site said "Math Overflow" with a space, as one would expect from a Stack Overflow analogue, this comment by Anton suggests that he thought of the site without a space from the very beginning, and in 2010 the design caught up.