Pick the simplest architecture that delivers measurable retention lift while staying within your app's size, latency, and moderation constraints; this article gives directional benchmarks, a 30-90 day sprint plan, and a quick A/B microtest with realistic effort and tradeoffs.
Core ML vs Cloud AI - Making the Right Call for Your Mobile App goes deeper on the ideas above and adds concrete next steps.
How do I choose on-device vs cloud vs hybrid? Concrete benchmarks
Category: Speed
Statistic: <150 ms
Label: Median on-device latency target
Context: Measure per device class with Systrace
Category: Speed
Statistic: P95 <300 - 400 ms
Label: Tail latency target
Context: Use P95 to catch worst-case UX regressions
Category: Efficiency
Statistic: 30 - 50 MB
Label: Quantized TFLite model footprint
Context: Keeps download size and storage pressure manageable
You can often triage on-device vs cloud vs hybrid in a short session if you already have representative device metrics and a few model candidates; otherwise expect a longer evaluation.
| Metric | Target | Why it matters |
|---|---|---|
| Median inference latency | <150 ms (modern mid-range devices) | Fast generation converts; measure with Systrace or adb profiling. |
| P95 latency | <300-400 ms | High tail latency damages conversion and retention. |
| Downloadable model size | 30-50 MB (quantized TFLite) | Keeps APK friction low; use Dynamic Feature Delivery for optional models. |
| Primary product signal | D7 retention lift among users who try generation | Direct business impact; base rollout decision on this. |
Explanation: measure median and P95 latency across representative devices, download size after quantization, and D7 lift from an early cohort.
Interpretation: if a quantized on-device candidate meets latency, size, and acceptable quality, it usually wins for conversion; otherwise cloud or hybrid are reasonable tradeoffs.
Reader impact: these quick checks let you pick a direction without committing months to the wrong architecture.
Risks, effort, and likely failure modes:
- Engineering effort: expect about 0.5-2 FTE for an MVP over 1-3 months; more work is common for training, scale, or custom ops.
- Moderation and ops: plan 0.5-1 FTE initially for triage and policy tuning; staffing scales with user volume.
- Infra cost: small pilots can be <$500/month, but per-request costs rise quickly as traffic scales.
- QA and device testing: budget several days to 2 weeks of device-farm testing; low-end devices often reveal latency surprises.
- Launch risks: Play Store reviews, rollback complexity, and moderation escapes are realistic; stage rollouts and have rollback plans.
Directional benchmarks (targets you can measure on your device cohort)
Aim for median <150 ms and P95 <300-400 ms measured across representative devices. Cap downloadable quantized TFLite binaries at about 30-50 MB and use Dynamic Feature Delivery for optional models. Define D7 retention lift up-front as the primary rollout decision metric.
Decision rule you can apply in 15 minutes
Prefer on-device when a quantized candidate fits your size cap, preserves user-rated quality, and meets tail-latency needs for conversion flows. Prefer cloud when quality materially improves UX and you accept extra latency and per-request cost. Use hybrid when a local fast-path handles common prompts and cloud handles complex or premium prompts. Note: this quick rule assumes you already have representative device metrics and labeled quality comparisons.
Early A/B microtest design (what to ship first and how to measure it)
Gate generation to 10-20% of active users and stagger by device tier to measure device-specific latency impact. Log generation_try, generation_complete, generation_latency_ms, generation_error, and link generation_try to cohort membership. Run until D7 estimates stabilize, typically 30-60 days for reliable D7 signals.
Sample-size guidance: for small effects (1-3 percentage points) expect several thousand users per arm; for medium effects (4-6 points) 1,000-2,000 per arm often suffices. If you lack variance estimates, aim for at least 1,000 users per cohort and stratify by device tier.
How do I ship a 30-90 day Android MVP - sprint plan and instrumentation
A three-row process diagram mapping Day 0 - 30 (model pick → TFLite conversion → NNAPI benchmarking), Day 30 - 60 (Remote Config gating → A/B instrumentation → hybrid fallback), Day 60 - 90 (analyze D1/D7 → scale decision → billing + Play Data Safety updates). Each row includes the key artifact to ship.
Ship a focused MVP in 30-90 days that produces latency and size numbers for your A/B test and a minimal moderation pipeline for safety; this is not a fully hardened production system.
Day 0-30: prototype, convert a candidate model, and verify latency
Pick and convert a candidate model
Select a small LLM or task-specific generator you've vetted for quality and convert it to TFLite.
Quantize and package
Apply post-training full-integer int8 quantization to target 30-50 MB and package as a Dynamic Feature Module or first-run download. Measure quality on a labeled prompt set because quantization can reduce fidelity.
Integrate and benchmark
Test with an NNAPI delegate and capture median and P95 with Systrace on a mid-range and a low-end device. Add an LRU cache for recent outputs and a minimal UI for trial. If quantized quality is unacceptable, plan for QAT, hybrid fallback, or an optional larger download.
Day 30-60: gated A/B rollout, fallback routing, and moderation pipeline
Remote Config A/B
Expose generation to treatment cohorts and log generation_try linked to cohort for retention analysis.
Fallback and hybrid routing
Add a cloud fallback for long or premium prompts, cache server responses, and monitor cloud tail latency and cost; implement throttles for spikes.
Client-side moderation
Ship a compact TFLite safety classifier to block or redact outputs before display; send flagged items to a server moderation queue with hashed IDs only and build an escalation path for human review.
How do I tune performance, moderate content, and meet Play Store requirements?
A compact checklist block listing Play Console tasks and moderation items: fill Data Safety form (processing location, model binaries), declare whether prompts leave device, add in-app reporting, test pre-launch for ANRs, add Remote Config gating, and confirm billing integration.
If early benchmarks pass, invest incrementally in tuning for scale, monitoring, and compliance; these improvements lower friction but add maintenance and observability costs.
Performance tuning and binary-size controls
Use int8 quantization and benchmark NNAPI vs CPU/GPU to see real median and P95 gains. Use Dynamic Feature Modules or Play Asset Delivery for optional downloads and include a tiny fallback model in the base APK. Cache recent generations with an LRU TTL and measure cache hit rate to understand user impact.
Content moderation, privacy, and minimal data retention
Run an on-device TFLite safety classifier to redact or block harmful outputs before display. Transmit only aggregated metrics or hashed prompt IDs for analytics and persist generated content server-side only when users explicitly save or share it. Maintain audit logs and retention windows aligned with your privacy policy.
Play Store Data Safety & launch ops checklist
Declare model binaries and whether prompts leave the device in the Data Safety form, disclose server-side processing, verify age ratings, add in-app reporting, and run Play Console pre-launch and device-farm tests. Stage rollouts and keep a rollback path.
