Groq runs open-weight language, speech and reasoning models on its own LPU hardware and NVIDIA clusters, serving them through an OpenAI-compatible API with per-token pricing, a free developer tier and thirteen data centres across four continents.
Used this tool? Rate it
Used this tool? Rate it
Groq is an AI inference cloud: infrastructure you send prompts to, not an application you log into and chat with. Groq describes itself as the premier neocloud for fast inference, combining infrastructure, inference and control in one platform, and the whole product narrative is built around one distinction — training produces a model, inference is what happens every time that model is actually used. The company puts it bluntly on its own homepage: Training creates the possibility, inference creates the value. Every customer served, every agent task completed, every commit reviewed by a model is an inference call, and as products lean harder on AI, that call becomes the bottleneck.
What makes Groq unusual among cloud providers is that it started at the silicon layer. The company pioneered the LPU — a processor designed specifically for running trained models rather than training them — and now operates that hardware alongside NVIDIA accelerated computing as a global production cloud. For a developer, all of that is hidden behind a single HTTPS endpoint. You point the OpenAI SDK at https://api.groq.com/openai/v1, swap the API key, and existing code runs on Groq without further changes. You are billed per token consumed, not per seat or per month.
The catalogue you get access to is deliberately open. Rather than hosting proprietary frontier models, Groq serves open-weight models — GPT-OSS, Qwen, Whisper and, historically, the Llama family — at speeds that closed-model APIs generally do not match. That trade is the central thing to understand about the product: you give up access to the very largest proprietary models and you gain latency, throughput and price predictability. Groq is a good fit when response speed is a product feature. It is the wrong fit when you need the single most capable model on the market regardless of how long it takes to answer.
LPU-based inference hardware: The Language Processing Unit is Groq's own silicon, purpose-built for inference rather than adapted from training hardware. Groq publishes rack-level figures of 256 LPUs per rack, 40 PB/s of SRAM bandwidth and 128 GB of on-chip SRAM per rack, alongside 315 PFLOPS of FP8 inference compute and a headline figure of 1,000 tokens per second per user. The architectural bet is that keeping model weights in fast on-chip memory, rather than streaming them from external HBM, is what removes the latency floor that GPU serving runs into.
An OpenAI-compatible API: Migration is deliberately trivial. Change the base URL, change the key, keep your existing OpenAI client library, and your application is running on different hardware. Groq also ships first-party Python and TypeScript SDKs for teams that prefer a native client. This single design decision is why Groq shows up so often as a drop-in speed upgrade in projects that were originally written against OpenAI.
An open-weight model catalogue: The production catalogue centres on open-weight models: openai/gpt-oss-120b and openai/gpt-oss-20b for text, plus whisper-large-v3 and whisper-large-v3-turbo for speech-to-text. Preview slots carry newer or more specialised models — Qwen3.6 27B, GPT-OSS Safeguard for moderation, Llama Prompt Guard for injection detection, and Orpheus voices for text-to-speech. Every model in the docs is published with its speed, price, rate limit, context window and maximum completion length, so you can cost a workload before writing code.
Compound: models plus built-in tools: Beyond raw model endpoints, Groq offers what it calls systems. Groq Compound is an AI system powered by openly available models that selectively uses built-in tools — including web search and code execution — to answer a query. Compound and Compound Mini run at roughly 450 tokens per second with a 131,072-token context window, giving you agentic behaviour without building the tool-calling loop yourself.
Speech-to-text as a first-class product line: Whisper is not a side feature here. It has its own pricing unit (per hour of audio rather than per token), its own rate-limit dimensions — audio seconds per hour and per day — its own 100 MB file ceiling, and dedicated transcription and translation endpoints. Teams building meeting notes, call analytics or subtitle pipelines can use Groq for the audio stage without adopting it for text generation.
Service tiers for different reliability needs: A single service_tier parameter selects how your request is scheduled. on_demand is the default and gives Groq's usual speed with occasional queueing at peak. flex trades reliability for headroom — ten times the rate limits at the same price, best effort. performance is the enterprise tier for latency-critical production. auto picks the best tier available to you at that moment.
Enterprise controls and dedicated capacity: The platform is sold as three stacked layers. GroqMetal provides dedicated bare-metal infrastructure, GroqCore adds the tested inference stack on top, and GroqAssured adds enterprise-grade governance, auditability and control — each layer including the ones below it. That structure lets a company start on the shared API and move to dedicated capacity without changing its integration.
Global footprint: Groq operates thirteen data centres across four continents, from Liberty Lake and Dallas to Vantaa, Sydney, London and Riyadh, with sites in Canada and Saudi Arabia alongside the US and European locations. For latency-sensitive applications, physical proximity to users matters as much as chip speed.
Real-time conversational interfaces: When a user is watching text appear, the difference between 50 and 500 tokens per second is the difference between waiting and reading. Support assistants, in-product copilots and voice agents are the clearest fit for Groq, because perceived responsiveness is the feature. Pairing a fast small model on Groq with a slower, more capable model for escalation is a common pattern: handle the ninety percent of queries that are simple at conversational speed, and route the rest elsewhere.
Agent loops and tool-calling chains: An agent that plans, calls a tool, reads the result and plans again multiplies latency by the number of steps. A five-step chain at two seconds per step is a ten-second wait; the same chain at 300 milliseconds per step feels immediate. This is where fast inference compounds most dramatically, and it is why Groq is frequently used underneath coding agents and workflow automations rather than for single-shot generation.
Bulk text processing and classification: Summarising a document backlog, tagging support tickets, extracting structured fields from thousands of records — these workloads care about throughput and unit cost rather than single-request latency. The flex tier exists precisely for this shape of work, offering ten times the standard rate limits at identical per-token pricing, and Batch processing handles jobs that can tolerate a delayed window.
Transcription and audio pipelines: Whisper on Groq handles meeting recordings, podcast archives, call-centre audio and subtitle generation, priced per hour of audio rather than per token. Because transcription and text generation live behind the same API and the same key, a pipeline that transcribes a call and then summarises it needs only one vendor relationship.
Prototyping and cost-controlled experimentation: The free tier requires no credit card and gives access to the model catalogue at reduced rate limits, which makes Groq a common first stop for hackathon projects, side projects and internal demos. Adding a payment method lifts the limits without committing to a subscription, so the cost of evaluating the platform is bounded by actual usage.
Migrating an existing OpenAI-based application: Because the API is compatible at the client-library level, teams often use Groq as a benchmark rather than a commitment — running the same prompts against both providers to compare latency, cost and output quality with a two-line configuration change. Some workloads move permanently, others stay split by task.
Create an account on the Groq Console. Sign up at the console, verify your email, and you land in an organisation — this matters, because quotas are tracked at the organisation level, not per user.
Generate an API key. Create a key in the console and store it as an environment variable such as GROQ_API_KEY. Never commit it to source control; the key authorises spending against your organisation.
Point your client at Groq. If you already use the OpenAI SDK, set base_url to https://api.groq.com/openai/v1 and pass your Groq key as api_key. If you are starting fresh, install the official groq package for Python or groq-sdk for TypeScript. A first request is a normal chat completion call with a model ID such as openai/gpt-oss-20b.
Choose the right model for the job. Read the supported models page before settling: it lists speed in tokens per second, price per million tokens, rate limits, context window and maximum completion tokens for each model. Prefer production models for anything you intend to ship — preview models exist for evaluation and can be withdrawn at short notice.
Set a service tier and handle rate limits. Leave service_tier unset for the default on-demand behaviour, or set it to flex for high-throughput batch-like work once you are on a paid plan. Read the x-ratelimit-remaining-tokens and x-ratelimit-remaining-requests response headers, and implement retry with jittered backoff on HTTP 429 and, for flex, HTTP 498.
Add billing controls before you scale. Attach a payment method to lift the free-tier ceilings, then set a spend limit and usage alerts in the console. Watch consumption under Dashboard → Usage during your first week in production, when estimates are least reliable.
Match the model to the task rather than defaulting to the largest. On Groq the smaller models are dramatically faster and cheaper, and for classification, extraction, routing and short-form generation the quality gap is often invisible to users. Benchmark gpt-oss-20b against gpt-oss-120b on your own prompts before assuming you need the bigger one — the speed difference is roughly two to one.
Design for model deprecation from day one. Groq's catalogue turns over quickly. Read the model ID from configuration rather than hard-coding it, keep an eye on the deprecations page, and make sure the email address on the account reaches someone who will act on migration notices. A model ID that disappears is an outage if your code cannot be redeployed quickly.
Do not try to buy headroom with extra API keys. Rate limits apply at the organisation level rather than per user or per key, so issuing extra API keys does not raise your ceiling. If you need more throughput, add a payment method, move to the flex tier or talk to sales about dedicated capacity.
Instrument first-token latency separately from total latency. Groq's advantage shows up in both, but they respond to different fixes. Slow first tokens usually mean queueing or a distant region; slow completion usually means the model is generating more than you need. Cap max_tokens aggressively — a shorter answer is faster on any hardware.
Use prompt caching and keep system prompts stable. Cached tokens do not count towards your rate limits, so a stable system prompt across requests both lowers cost and buys you effective throughput. Rewriting the system prompt on every call throws that away.
Retry properly, especially on flex. The flex tier is explicitly best-effort: capacity failures are expected, arrive quickly, and are safe to retry. Jittered exponential backoff is not optional there. Treat HTTP 498 as "try again shortly", not as an error to surface to users.
Turn on Zero Data Retention before sending anything sensitive. ZDR is available to all customers but is not the default, and enabling it also disables features that need persistence. Decide which side of that trade you are on before your first production request, not afterwards.
Keep an escalation path to a different provider. Because the API is OpenAI-compatible in both directions, keeping a second provider configured behind the same interface costs little and protects you against capacity events, deprecations and pricing changes.
Application developers building latency-sensitive AI features: Anyone whose users watch text appear — chat interfaces, copilots, voice products — gets the most direct benefit, because speed converts straight into perceived quality.
Teams building agents and multi-step workflows: When latency multiplies across steps, fast inference is worth more than it looks on a single-request benchmark. Agent frameworks, coding assistants and automation pipelines are natural fits.
Engineers already invested in the OpenAI SDK: If your codebase is written against OpenAI's client libraries, evaluating Groq costs a configuration change rather than a rewrite. That low switching cost is the platform's most effective acquisition channel.
Cost-conscious teams running high volumes of simple tasks: Classification, tagging, extraction and summarisation at scale are far cheaper on small open-weight models than on proprietary frontier ones, and Groq's per-token pricing makes the arithmetic straightforward.
Startups and prototypers: The free tier without a credit card, plus pay-as-you-go with no subscription floor, makes the platform easy to try and easy to abandon — which is exactly what an early-stage team wants.
Enterprises with latency, governance or residency requirements: GroqAssured, the performance service tier, Zero Data Retention and dedicated bare-metal capacity exist for organisations that cannot run on a shared best-effort endpoint.
Teams building audio and transcription products: Whisper pricing per audio hour, dedicated audio rate limits and translation endpoints make Groq viable as a transcription vendor independently of whether you use it for text.
https://api.groq.com/openai/v1, covering chat completions, the Responses API, audio transcription, translation and speech, batches, files and fine-tuning endpoints.console.groq.com provides account management, API keys, projects, model permissions, usage dashboards, spend limits and data controls, along with the full documentation and API reference.Free tier. Groq offers a free developer tier that requires no credit card and provides access to the model catalogue at reduced rate limits. It is genuinely usable for prototyping, side projects, internal demos and low-volume features, and the constraint you hit first is usually requests per day rather than tokens. The important structural detail is that limits are tracked per organisation, so a free account cannot be widened by creating more keys.
Pay-as-you-go. There is no subscription fee for API access. Text models are priced per million tokens with separate input and output rates — for example, gpt-oss-120b at $0.15 input and $0.60 output per million tokens, and gpt-oss-20b at $0.075 and $0.30 — while speech-to-text is billed per hour of audio, with whisper-large-v3 at $0.111 per hour and the turbo variant at $0.04. Adding a payment method raises rate limits substantially and unlocks the flex and batch processing tiers, which offer higher throughput at the same per-token price. Independent benchmarking by Artificial Analysis puts the blended range across Groq's catalogue at roughly $0.05 to $0.84 per million tokens depending on model, a spread of about sixteen times.
Billing mechanics. Charges are progressive rather than purely monthly: the first crossings of the $1, $10, $100, $500 and $1,000 lifetime thresholds trigger immediate charges before the account settles into monthly invoicing, with a different schedule for customers in India. Invoices under $0.50 are not issued. Spend limits and budget alerts are available in the console, and usage is visible in near real time.
Enterprise. Committed-spend contracts, the performance service tier, dedicated GroqMetal capacity and the GroqAssured governance layer are sold through sales rather than self-service. Enterprise contracts also carry practical benefits beyond capacity — committed-spend customers are exempted from the model deprecations that apply to free and developer-tier usage. Exact terms and current rates should be confirmed against the official documentation and console, since the catalogue and its pricing change frequently.
The model catalogue turns over aggressively. This is the single most practical risk. Groq's own deprecations page documents a steady stream of retirements: llama-3.1-8b-instant and llama-3.3-70b-versatile were both retired on 16 August 2026, with qwen3-32b and llama-4-scout removed a month earlier and Kimi K2 and Llama 4 Maverick before that. Deprecation applies to free and developer-tier usage while committed-spend enterprise customers are exempt, which means the least-protected users absorb the most churn. Preview models carry an explicit warning that they may be discontinued at short notice.
OpenAI compatibility is close but not complete. Several parameters that appear in OpenAI-targeted code will fail outright: logprobs, logit_bias, top_logprobs and messages[].name all return a 400 error, and n must equal 1. Audio transcription does not emit vtt or srt, and a temperature of 0 is silently rewritten to 1e-8. Migration is easy in the common case and can still break on the edges, so test rather than assume.
Capability is capped by the open-weight catalogue. Speed is real but it is speed on mid-sized open models. In Artificial Analysis's independent benchmarking, the highest Artificial Analysis Intelligence Index score among the eleven Groq models it tracks belongs to Qwen3.6 27B at 38 — well below proprietary frontier models. For the hardest reasoning, long-horizon coding or nuanced writing tasks, a slower frontier model may still produce better results.
Rate limits are organisational and multi-dimensional. Requests per minute, requests per day, tokens per minute, tokens per day and audio seconds all apply simultaneously, and you are throttled by whichever binds first — a pattern of many tiny requests can exhaust RPM while barely touching your token allowance. Some organisations additionally have separate input and output token limits.
The flex tier is explicitly best-effort. It offers ten times the rate limits at the same price, but requests fail with HTTP 498 and a capacity_exceeded error when capacity is unavailable. That is a deliberate design, not a fault, and it means flex is unsuitable for user-facing paths without a fallback.
Data residency is fixed to the United States. Although inference requests are not retained by default and Zero Data Retention is available to all customers, any data that is retained lives in Google Cloud Platform buckets located in the United States, with standard contractual clauses covering cross-border transfer. Organisations with strict EU or APAC residency requirements should confirm this against their compliance obligations, and note that enabling ZDR disables features that depend on persistence, such as batch jobs and fine-tuning.
The corporate situation has recently been turbulent. In December 2025, DataCenterDynamics reported that founder Jonathan Ross and president Sunny Madra left for NVIDIA under a non-exclusive technology licensing agreement while Groq continued as an independent business under Simon Edwards, with GroqCloud operating uninterrupted. Reporting on the deal's value has been unstable: the widely repeated $20 billion figure has not been independently verified, and the terms of the non-exclusive licence were never disclosed. The August 2026 Series A valued the company at $3.5 billion, roughly half the $6.9 billion valuation at which it raised in September 2025. Service continuity has been maintained throughout, but teams making multi-year infrastructure commitments should weigh the ownership and leadership changes alongside the technology.
Independent user review data is thin. Groq has extensive press coverage but very little structured review-platform feedback — the G2 listing carries a negligible number of reviews and third-party directories classify Groq under AI API, large language models and open-source AI models rather than as a consumer chatbot, with no meaningful review volume. The strongest independent evidence available is benchmark data rather than user testimony, so claims about support quality or long-run reliability should be treated as unverified.
Not to be confused with Grok. Groq (groq.com) is inference infrastructure from Groq LLC. Grok (grok.com) is a consumer chat assistant from Elon Musk's xAI. The names differ by one letter's position and the products share nothing — different companies, different audiences, different business models.
Yes, there is a free developer tier that requires no credit card and gives access to the model catalogue at reduced rate limits, which is enough for prototyping and low-volume projects. Adding a payment method raises those limits substantially and unlocks the flex and batch tiers; there is no subscription fee, only per-token usage charges.
They are unrelated products that are frequently confused because of the near-identical spelling. Groq, at groq.com, is an AI inference cloud built on LPU hardware and sold to developers as an API. Grok, at grok.com, is a consumer chat assistant from xAI. Different companies, different products, different pricing models.
In most cases yes, with two changes: set the base URL to https://api.groq.com/openai/v1 and supply a Groq API key. A few parameters are unsupported and will return a 400 error — logprobs, logit_bias, top_logprobs and messages[].name — and n must be 1, so test your specific call patterns rather than assuming a clean swap.
Groq serves open-weight models rather than proprietary frontier ones. The production catalogue centres on openai/gpt-oss-120b and openai/gpt-oss-20b for text and whisper-large-v3 and its turbo variant for speech, with preview slots for models such as Qwen3.6 27B, safety classifiers and Orpheus text-to-speech voices. The current list is always available from the models documentation and the /openai/v1/models endpoint.
Independent benchmarking by Artificial Analysis puts the fastest model on Groq, gpt-oss-20b at high reasoning effort, at a median 949 tokens per second, with the lowest time to first token around 0.77 seconds. Groq's own hardware figures cite up to 1,000 tokens per second per user. Actual throughput varies with model, prompt length, service tier and load.
Groq states that it does not retain customer data for inference requests by default. Data is retained only for features that require persistence, such as batch jobs and fine-tuning, or temporarily for reliability troubleshooting and abuse investigation, in which case it is kept for up to 30 days. All customers can enable Zero Data Retention, though doing so disables the features that depend on persistence.
Inference runs across thirteen data centres on four continents, but any retained customer data is stored in Google Cloud Platform buckets located in the United States, with standard contractual clauses covering transfers where applicable. Organisations with strict regional residency requirements should verify this against their own compliance rules.
No. Rate limits apply at the organisation level, so additional keys share the same quota. To get more headroom, add a payment method to move to developer-tier limits, use the flex tier for high-throughput work, or arrange dedicated capacity through sales.
Usage is billed per token for text models and per hour of audio for transcription, with no subscription component. Charges are progressive at first: the account is billed when lifetime usage crosses $1, $10, $100, $500 and $1,000, after which billing becomes purely monthly. Invoices below $0.50 are not issued, and spend limits with budget alerts are configurable in the console.
GroqCloud has operated continuously through significant corporate change, including the NVIDIA licensing agreement, the departure of the founder and a leadership transition, and the company raised a $350 million Series A in August 2026 at a $3.5 billion valuation. The practical risk is less about the company disappearing and more about model churn: build with configurable model IDs, monitor deprecation notices, and keep a second provider available behind the same interface.