A smaller model download and an impressive benchmark can make a quantized checkpoint look like an obvious upgrade. On another machine, it may simply make the model fit. Responses might not get faster at all.
File size, runtime GPU memory, and user-visible latency are separate outcomes. A “4-bit” label does not guarantee all three.
For an RTX 3090 deployment, I would investigate the storage format, the selected kernel, and the actual workload before accepting a speed claim. This article provides that investigation process, not benchmark results from a dual-3090 system. The documentation was checked on September 25, 2026; verify support against the release you intend to deploy.
Bit width does not specify the execution path
Quantization methods, numerical formats, file containers, and execution kernels answer different questions.
AWQ and GPTQ are commonly used weight-quantization methods. A checkpoint still has particular bit-width, grouping, and storage choices. NVFP4 is a low-precision floating-point representation. Their appearance in the same menu does not put them at the same conceptual level. A container such as GGUF does not identify its precise quantization type by filename extension alone, either.
The activations matter too. W4A16 indicates 4-bit weights with 16-bit activations; W4A4 indicates 4-bit weights and activations. Neither shorthand fully specifies the precision of every operation or accumulation.
When moving weights is a substantial part of the workload, moving fewer bytes can help. Unpacking, dequantization, matrix operations, and other work still cost time. A compute-heavy workload may save too little memory traffic to compensate for extra processing.
Input length, batch size, and concurrency change that balance. The useful question is not just whether a model is 4-bit, but which implementation executes it in the chosen framework.
No native FP4 does not mean no possible FP4 execution
NVIDIA lists the RTX 3090 at compute capability 8.6, commonly written as SM 8.6. It is an Ampere GPU. Blackwell's native NVFP4 acceleration cannot simply be transferred to it.
NVIDIA's NVFP4 introduction explains the corresponding Blackwell hardware and grouped scaling design. This concerns how low-precision values are represented and processed, not just how little disk space they occupy.
It would still be too broad to conclude that a 3090 cannot run any FP4 checkpoint.
The current vLLM documentation includes a Marlin implementation for NVFP4 weights. It uses W4A16. That offers a weight-compression execution path without turning the GPU into native FP4 hardware. The implementation explicitly warns that compute-heavy workloads can lose performance on this path.
That is a conditional capability, not a universal compatibility promise. A particular checkpoint, layer structure, or framework release may still be unsupported. Use the vLLM compatibility matrix to shortlist possibilities, then verify the model configuration and the implementation actually selected at startup.
Loading successfully establishes that one hurdle has been cleared. It does not establish lower latency or a useful speedup.
Two 24 GB cards are not a single 48 GB allocation
Parameter count multiplied by bit width gives a rough starting point for weight storage. For a hypothetical eight-billion-parameter model, assigning exactly four bits to every parameter gives four billion bytes: 4 GB, or about 3.73 GiB.
That deliberately simplified calculation is not a real checkpoint size and certainly not the complete GPU memory requirement. Scales, other quantization metadata, and components retained at higher precision change the total.
Runtime memory also includes activations, temporary workspaces, and the KV cache used to retain attention information during generation. Longer contexts and more simultaneous requests can increase pressure; the details depend on the architecture and implementation.
With two cards, distribution depends on the model's parallelization strategy. One GPU can exhaust its memory even when the aggregate appears sufficient. Splitting work also introduces communication. Inspect the actual links and topology rather than assuming that a supported interconnect is installed or being used.
A model that fits on one GPU may not serve a single request faster on two. The second card might be more valuable for capacity or concurrent requests. Record peak memory on each GPU rather than reporting only a combined total.
“CUDA version” is not one compatibility check
The driver, the system CUDA Toolkit, and the CUDA runtime associated with a PyTorch build are separate layers. The CUDA figure displayed by nvidia-smi is not proof that a matching Toolkit is installed. A CUDA suffix in a PyTorch package name does not replace the driver check.
NVIDIA's current minor-version compatibility documentation gives a minimum driver family of 525 for CUDA 12.x and 580 for CUDA 13.x. Minor-version compatibility has limitations; newer features or PTX execution may need a newer driver. Meeting the family minimum is not a guarantee for every framework package.
Suppose an older environment has a 535-series driver and an older PyTorch build. Installing a new package that requires CUDA 13 and then blaming the failure on the 3090's quantization support conflates several separate issues.
Keep the working environment intact and test a candidate stack separately. Record the model revision, framework release, PyTorch build, driver, and launch arguments. A Python environment does not isolate the host GPU driver. A container cannot add hardware capabilities that do not exist.
Store model files in the project's ./models directory, and prepare local copies of required packages and dependencies for an internal-network deployment. There is no generally safe “upgrade everything” command independent of the selected model and framework.
Compare the same work, not attractive numbers
Prepare a short-input request, a long-input request, and a concurrent request set. These probe different mixtures of token generation, input processing, and service load. None predetermines which quantization option should win.
Keep the base model, input content, input and output lengths, sampling settings, and service configuration aligned as far as possible. Warm up before timing, repeat the runs, and document caching. Comparing a prefix-cache hit with a cold request measures more than the quantization change.
For each configuration, record:
- Model and quantization settings, software releases, selected kernel, and GPU count.
- Input and output token counts, concurrency, context limits, warmup, and caching.
- Time to first token, measured from sending the request to receiving its first output token.
- Subsequent generation rate, aggregate throughput, and peak memory on each GPU.
- Variation across runs, errors, and answer-quality checks.
These measurements serve different purposes. Higher aggregate throughput can coexist with a longer wait for the first token. Short-input results may not describe document processing. State whether latency includes queueing and network time so that comparisons use the same definition.
Check output quality on verifiable public or fictional inputs. Information extraction, preserving quantities, and following required formats are useful when those are the actual jobs. Producing fluent prose is an insufficient acceptance test by itself.
If the higher-precision model cannot fit on the same hardware, report the result as enabling that model to run on the device. A comparison against a larger-memory machine can still be informative, but disclose the hardware differences rather than assigning the entire speed difference to quantization.
Decide what to download with a deployment checklist
For a 3090, I would start with candidates that have documented support for the intended stack, rather than ranking formats by novelty. Whether AWQ, GPTQ, or another option is appropriate depends on the actual model and framework release.
Before downloading, check the architecture, license, quantization configuration, and framework support. After startup, check the selected kernel. Under load, check capacity, latency, and answer quality against the intended use. Fix driver or loading problems directly instead of hiding them by choosing a smaller model.
When asking for help, include the model location and revision, quantization settings, GPU topology, driver and framework releases, complete launch arguments, and a sanitized error. Those details make “why is 4-bit slow?” an answerable question.
Smaller weights can be useful on their own. If the objective is lower latency, measure that separately. If the objective is to fit a model that otherwise cannot run, there is no need to chase a speedup reported for an unrelated setup.
For a discussion of a specific deployment, bring that environment information and a minimal reproduction to my GitHub. Without comparable measurements, I would not label any format the best choice for every 3090 installation.