[I just got official okay to mention this in public. This was previously announced at Apple's WWDC conference last week.]
For those who are interested, Apple announced that they are using the LLVM optimizer and JIT within their Mac OS 10.5 'Leopard' OpenGL stack (which was distributed in beta form to WWDC attendees).
LLVM is used in two different ways, at runtime:
1. Runtime code specialization within the fixed-function vertex-processing
pipeline. Basically, the OpenGL pipeline has many parameters (is fog
enabled? do vertices have texture info? etc) which rarely change:
executing the fully branchy code swamps the branch predictors and
performs poorly. To solve this, the code is precompiled to LLVM .bc
form, from which specializations of the code are made, optimized,
and JIT compiled as they are needed at runtime.
2. OpenGL vertex shaders are small programs written using a family of
programming langauges with highly domain-specific features (e.g. dot
product, texture lookup, etc). At runtime, the OpenGL stack translates
vertex programs into LLVM form, runs LLVM optimizer passes and then JIT
compiles the code.
Both of these approaches make heavy use of manually vectorized code using SSE/Altivec intrinsics, and they use the LLVM x86-32/x86-64/ppc/ppc64 targets. LLVM replaces existing special purpose JIT compilers built by the OpenGL team.
LLVM is currently used when hardware support is disabled or when the current hardware does not support a feature requested by the user app. This happens most often on low-end graphics chips (e.g. integrated graphics), but can happen even with the high-end graphics when advanced capabilities are used.
Like any good compiler, the only impact that LLVM has on the OpenGL stack is better performance (there are no user-visible knobs). However, if you sample a program using shark, you will occasionally see LLVM methods in the stack traces. 
-Chris