If you don't distribute source code, then what you are distributing cannot meaningfully be called "Open Source". And if you don't distribute at all, then by definition you're not distributing source code, so you're not distributing anything Open Source.
Think of it this way: Open Source licenses are always applied to the source code — so if you're not distributing the source, then you're not distributing the thing to which an Open Source license applies. You might or might not distribute binaries; that's a separate question. But while some Open Source licenses allow you to distribute binary code without distributing the corresponding source, it is only the source code that can be "open source". The binaries alone cannot be Open Source, because you're not making any source code available to be open. (If someone else distributes the source code under an Open Source license, then that's still Open Source, of course.)
Note that copyleft Open Source licenses require redistributors to make source code available under certain circumstances; for example, see the GNU General Public License and GNU Affero General Public License.