The easiest starting point is to get Clojure up and running with Leiningen. Leiningen is an essential Clojure tool that can create projects, resolve library dependencies, compile your code, run your tests, fire up a REPL and a whole lot more. You need Leiningen - and you can get away with just that and a text editor if your tooling needs are simple!
Cake is another Clojure build tool, very similar to Leiningen, that uses the same project file format and offers a lot of the same functionality. Whilst Leiningen is more popular, a lot of people like Cake. Another tool for running Clojure code that's worth checking out is Jark which, like Cake, uses persistent JVMs to reduce startup time.
According to Chas Emerick's 2011 state of the Clojure union survey, the vast majority of Clojure developers use Emacs. I used Emacs years ago - 20+ years ago - it was great. Emacs has evolved but, well, it's still Emacs. Since those days, I've worked with Java a lot and gone thru a number of Java-centric IDEs and I've gotten very used to them (anyone remember TogetherSoft? I loved that IDE!). So my natural IDE of choice for Clojure was Eclipse + Counter ClockWise. I like the "structural" mode that makes it easy to keep parentheses balanced and provides keyboard shortcuts focused on manipulating s-expressions - the building blocks of Lisps - in other words, CCW brings Emacs' paredit mode to Eclipse. CCW also provides an integrated REPL, syntax highlighting, "rainbow" parentheses coloring and a namespace browser, as well as code checking based on being able to compile your Clojure code as you type. CCW is actively maintained (by Laurent Petit and several othes, including Chas Emerick). If Eclipse is your drug of choice, CCW will be a good way to work with Clojure, especially if you're building polyglot projects (as we are at World Singles).
Other Java IDEs also have Clojure plugins (Enclojure for NetBeans, La Clojure for IntelliJ) and various text editors have support for Clojure as well, either built-in or via a bundle / plugin. I've tried TextMate and Sublime Text 2 and, whilst I like the lightweight aspect of those, their Clojure support is still quite immature. There's also a promising new project called Clooj that provides a standalone editor for Clojure - all written in Clojure.
Despite all these options, Clojure tooling is still pretty young and has a lot of rough edges, with only fledgling support for debugging. How much this will matter to your impressions of Clojure will depend on how much you rely on sophisticated IDE tooling in your current language - but given how much simpler Clojure is than most languages, you may not miss various features as much as you might expect!
At World Singles, we use Leiningen for dependency management, running tests, running individual "main" programs (for cron jobs and some long-running daemon processes), as well as starting a REPL in the context of our project. We run Leiningen as needed from our ant build script (using the exec task) but we haven't yet fully integrated our Clojure tests into our regular JUnit tests because we haven't managed to get the Leiningen test-out plugin working correctly.
As a die-hard Eclipse user, I was very happy with CCW on my powerful desktop system but I wanted something lighter in weight on my lower-powered netbook. I finally took the plunge and installed Emacs on both my desktop Mac and my Ubuntu netbook. There are a lot of conflicting articles out there about getting Emacs up and running for Clojure and in the end it took me about five hours of pain and frustration to get both machines set up. As of today, I've been full-time in Emacs for a week now and, whilst it really does have a steep learning curve (and some bewildering keyboard shortcuts, compared to what we've become used to in our GUI-driven lives, over the last decade), the more you use it, the more powerful you realize it is. I've settled on a layout where about two thirds of my screen is my file editor, with a #clojure IRC session open in a buffer, top right, a REPL bottom left and a shell session bottom right. I'm still relying a lot on the built-in help system but the "apropos" search usually helps me find what I need and I'm slowly memorizing the keyboard shortcuts... Given that Emacs is programmed in Lisp, I can see why anyone using a Lisp would gravitate toward it.
To learn Clojure and get up and running, I'd recommend sticking with your current, preferred IDE, installing the appropriate plugin and using Leiningen to manage projects. Once you're comfortable with that, consider Emacs - the Clojure community is very helpful at getting new folks up and running and full of useful hints and tips.