Model Limits & Critique
A structured AI critique and response — what the model gets right, what it doesn’t, and what we’re doing about it
September 17: Read the discussion responses to Reports 5 and 6, implemented alternatives, provenance, and open questions. The original September audit remains a record of that version.
Earlier September review and fixes: Read the code audit, corrected source comparison, and priorities for the next model. Both browser views now share one tested calculation engine. Utilization accounting, complete-medium double counting, probability endpoints, and several misleading explanations have been corrected. Physical feasibility and calibration remain open.
Output basis: changed to dry cell mass. Since engine 2026-10-26.4 the headline cost is per kg of dry cell mass, with cost per kg protein and a chicken-meat equivalent (dry cost × 27.5% placeholder solids) alongside; thresholds and blending use the meat equivalent. Per wet kg is not a stable unit (washing and dewatering change what it contains; a paste at 15% solids carries half the protein of one at 30%), while dry mass is invariant to that, maps to nutrition and suits nutrient stoichiometry. Density is entered in g dry cells per litre. The old wet-basis output is one switch away (cost_basis: "wet") and reproduces earlier results exactly; default dry figures are about five times the old wet ones. See Target quantity and why the basis matters.
Target (estimand). Unconditional and hypothetical: the cost of a large (“nth-of-a-kind”) plant built with the technology available in 2036, whether or not such plants exist; the best available plant, not the industry average; 2026 US dollars; no retail-price modeling. See Estimand.
AI-drafted (Claude, 26 October 2026) from David Reinstein’s design decisions; not yet reviewed by a person.
Source of this review: This critique was generated by prompting ChatGPT Pro (deep research mode) with the model page, beliefs form, workshop site, and uploaded context documents in early 2026. It was not written by a human expert reviewer. We share it because it raises substantive methodological points worth addressing — but treat it as a structured AI critique, not a peer review.
The full prompt and response are on GitHub: CM_model_deep-research-report.md.
The Unjournal team’s assessment: We’ve been reviewing this critique and so far it generally holds water. The structural issues it identifies — output definition inconsistency, the ad hoc correlation structure, sensitivity analysis caveats, and beliefs-form alignment — are genuine and worth addressing, and we have acted on several of them. A few suggestions appear to have overlooked elements we’ve already incorporated into the model. Other suggestions go beyond our current bandwidth, but we’d like to follow up on them given the right resources, expertise, and participant willingness. We’ll be asking workshop participants directly about their interest in contributing to these improvements.
The reviewer’s bottom line:
“The project is already good enough to support productive discussion, but not yet good enough to serve as the authoritative modeling-and-elicitation backbone for coordinated expert convergence.”
We agree. This model is designed to support structured exploration and deliberation — not to produce a single authoritative posterior. The limitations below are real, and we name them rather than paper over them. Both the model and beliefs form have been updated since this review was generated — some issues are already addressed, others are work in progress.
What the model and process already do well
- Public and transparent — code, formulas, and parameter choices are all exposed and annotatable via Hypothesis
- Uncertainty-aware — Monte Carlo simulation with explicit distributions rather than point estimates
- Explicitly caveated — documentation notes the static snapshot structure, missing geography, and ad hoc dependence
- Linked to deliberation — designed for a workshop where beliefs are elicited before and after evidence review
- Discussion-connected — GitHub Discussions, Hypothesis annotations, and participant forms feed back into the model
Key structural issues in the model
Implementation consistency
Browser mismatch corrected in September 2026. Simple and Advanced previously used different calculations, despite claiming equivalence. They now share cost-model.mjs, with tests that compare their actual parameter adapters at matching settings. The JSON export includes the engine version and effective simulation parameters.
Python and Squiggle remain historical. They preserve older assumptions and must not be used as current equivalents of the dashboard. Current numerical audits run the shared JavaScript engine directly. This removes the need to maintain an independently evolving Python calculation just to reproduce browser results.
Target quantity: wet biomass, solids content, and the harvest gate
Issue. The model intends to estimate pure cultured chicken cell biomass around the harvest gate, before product formulation. But “wet kg” is not a stable accounting unit unless water content, residual media, washing, dewatering, dry matter, and protein content are standardized. The code treats cell-density inputs as grams of wet biomass per litre; it does not separately apply the 80% hydration assumption previously stated in the interface. Harvested biomass is also not automatically identical to an edible finished ingredient: recovery and washing losses may sit between the bioreactor and the usable input.
Status: Largely addressed; dry basis adopted (26 October 2026). David Reinstein revised the decision the same day: kg of dry cell mass is the primary internal basis for costs and yields, with cost per kg protein as a secondary output and a chicken-meat equivalent (dry cost × a fixed solids content, placeholder 27.5% within 25–30%) for intuition; cell number is not a cost basis. Engine 2026-10-26.4 implements this as the default. Density inputs are g dry cells per litre; $/kg inputs are per kg dry; the nutrient-yield option is in g dry cells per litre of medium. Wet-basis sources, the built-in wet-basis priors and plant capacity are converted with an explicit solids fraction (20%), and the documentation records how sources report their basis (including the ambiguous basis of Humbird’s headline $/kg). The wet-basis output stays available behind cost_basis: "wet" until the change is reviewed. Still to do: agree the meat-equivalent solids content; source the solids and protein shares and the harvest cost (a process model is the planned source); add an uncertain mass-per-cell parameter for cell-count sources (per-cell mass varies about twofold with growth rate and phase, and published models may overestimate it about threefold, per GFI’s 2026 TEA 2.0 Phase 1a review); add a yield per gram of glucose or amino acids; and model washing and dewatering losses (downstream scope).
Status note AI-drafted (Claude, 26 October 2026); not yet reviewed by a person.
Conditional success versus probability of commercialization
Issue (as raised). CM_01 asked for cost in 2036 assuming large-scale commercial production is achieved. The model, however, samples low-maturity/high-cost worlds without separately modeling the probability that a qualifying commercial plant exists. It therefore mixed two different uncertainties: whether commercialization occurs and what cost is conditional on occurrence. It also left ambiguous whether the target is an average commercial plant, a capacity-weighted sector average, or the frontier/best feasible plant.
Status: Resolved by reframing the target (26 October 2026); hurdle model kept as an option. The team decided that the model estimates an unconditional, hypothetical quantity:
- the cost of a large (“nth-of-a-kind”) plant built with the technology available in 2036, whether or not such plants exist;
- the best available plant, not the industry average;
- in 2026 US dollars;
- with no retail-price modeling.
Because the target no longer conditions on commercialization happening, the model does not need a separate probability-of-commercialization stage. The hurdle structure recommended earlier, (1) probability that qualifying large-scale capacity exists by the target date, (2) cost conditional on that event, and (3) capacity-weighted output, is no longer the default path. It remains an optional extension, for example to answer CM_01 parts A2 (conditional actual cost) and B (probability of production) below.
Caveats on the reframed target.
- The 2036 technology level is not exogenous. What technology is available in 2036 depends on investment and learning between now and then, which in turn depend on how well the industry does commercially. An unconditional “technology available in 2036” still embeds beliefs about that path.
- First-of-a-kind plants typically cost more. The early plants actually built in or before 2036 are likely to cost more than the nth-of-a-kind plant this target describes, because of design margins, commissioning problems and lack of learning. The model’s output should not be read as the cost of the first commercial plants.
Alignment with CM_01. The CM_01 beliefs question was reworded on 10 September 2026 and is now unconditional and hypothetical as well (question changes). Its central part, A1, reads: “What will be the production-weighted average cost per kg of cultivated-chicken cell biomass in 2036, at a scale of at least 2,000 tonnes per plant per year?”, with the instruction “If you imagine no plants producing at this scale, consider the hypothetical cost at this scale.” Conditional actual cost (A2) and the probability of production (B) are now separate questions. Three differences between A1 and the model’s target remain open:
- Best plant versus average. A1 asks for a production-weighted average and says “Do not use the cheapest plant or a single representative plant”; the model targets the best available plant. Comparisons between model output and A1 responses should treat the model as closer to a lower bound.
- Dollar year. A1 uses 2025 US dollars; the model’s target is 2026 US dollars (and its inputs are not yet normalized to either; see below).
- Weight basis. A1 uses wet weight with 80% water as its reference; the model now reports per kg dry cell mass. Compare via the model’s wet basis (
cost_basis: "wet"), or multiply A1 answers by 5 to put them per kg dry.
The CM_01 Metaculus question keeps its older wording: the average production cost “across all large-scale plants in the world”, wet weight at harvest, 2025 dollars, resolved by asking an AI model about current costs. That is an actual-outcome average, so it is neither the unconditional hypothetical framing nor the best-plant target. We have not changed it.
Parity reference. The team hub’s parity reference for conventional chicken (about $9–10/kg) looks like a retail-type price. A factory-gate production cost should be compared with conventional chicken at a comparable point in the supply chain (for example, processor or wholesale cost per kg of edible meat), as CM_01 part C now specifies.
This distinction also matters when comparing the model with the workshop beliefs and Pablo AMC’s more frontier-oriented interpretation of the TEAs.
Status update AI-drafted (Claude, 26 October 2026) from David Reinstein’s decisions; not yet reviewed by a person.
October 2026: an opt-in least-cost mode selection uses the cheapest feasible process mode in each world, with feasibility probabilities in place of mode weights. That fits the best-available-plant target, but for the process choice only; other ranges are not yet frontier-specific. Until mode-specific equipment, consumable and contamination costs are modeled, it favors whichever mode has the fewest downsides represented.
Currency year and cost-gate consistency
Issue. Inputs drawn from different publications and years are not yet normalized to one constant-dollar base year. The demand bridge then compares a future factory-gate biomass-cost scenario with approximate 2025 conventional retail prices. Even perfect inflation adjustment would not solve the factory-gate-versus-retail mismatch.
Status: Open and high priority; base year chosen. The target is now 2026 US dollars (decided 26 October 2026; CM_01 A1 uses 2025 dollars). Display it; document the deflator for every imported monetary input; and add a separate value-chain bridge covering inclusion rate, recovery yield, non-cell ingredients, downstream processing, packaging, distribution, tax, and retail margin before making price-parity comparisons.
Supplemental recombinant proteins (albumin, transferrin, insulin)
Implemented, but weakly grounded. Separable mode includes a supplemental-protein cost, with p5–p95 $0.03–0.60/kg in its low-cost regime and $0.50–4.00/kg in its higher-cost regime. These are provisional direct cost priors. They do not calculate concentration, consumption, or replacement requirements for each protein.
Complete-medium mode now includes all these proteins within its media cost and zeros separate protein lines. The next scientific improvement is a formulation-specific quantity and price model; see the September review.
Dependence structure: the single “maturity” factor
Issue. The model uses one latent “maturity” variable to correlate technology adoption rates, financing costs (WACC), and reactor choice. This makes favorable assumptions co-occur, but asks one synthetic dimension to stand in for sector maturity, supply-chain development, regulatory learning, and financing all at once — creating potential hidden coupling and double-counting.
Better practice would use at least three separate latent factors:
- Technical bioprocess maturity
- Supply-chain / input maturity
- Financing / regulatory maturity
…with user-selectable correlation among them, making the dependence structure itself an object of scrutiny.
Status: Planned. Multi-factor dependence is on the roadmap. A first step: offer three correlation modes — independent, sector-coupled (current), and strongly coupled — so users can see whether headline results are robust to the assumption.
Parameter grounding: empirically informed but not always documented
Issue. The model’s parameters draw on published TEA literature, GFI research, and supplier data — but the linkage from source evidence to distribution shape is not always made explicit. A reader cannot always tell whether a given distribution reflects a specific study, a range across studies, or a calibration judgment.
Status: Ongoing. We reference key sources where we can (GFI amino acid report, Pasitka et al., Humbird 2021, Rethink Priorities). More systematic documentation of evidence–distribution mappings is on the roadmap. In the meantime, the beliefs form invites expert participants to provide their own distributions, which will be compared against model defaults.
A specific case worth flagging: basal media cost ($/L) and cell density (g/L) are treated as independently sampled parameters in the model, but they may be systematically related. In nutrient-limited fed-batch systems, higher cell density requires proportionally richer — and more expensive — media formulations, because density is ultimately bounded by the grams of nutrients cells can extract from a given volume. Combining very low \(/L with very high density in the same scenario may therefore produce unrealistically optimistic cost estimates. This coupling is strongest in fed-batch mode and weaker in perfusion (where fresh media is continuously supplied). A fuller model would sample these parameters with explicit correlation structure; for now, users should treat highly optimistic joint density-and-\)/L combinations with caution. The beliefs form deliberately asks for $/kg biomass (not $/L separately) to allow respondents to integrate this interaction themselves.
Sensitivity analysis caveats
The tornado chart is a useful exploration tool, but it has well-defined limitations.
Limitation. The chart shows how much the mean cost changes when each parameter moves from its 10th to 90th percentile — a conditional-mean dollar-swing statistic. This is not a Sobol variance decomposition: bars for correlated parameters can overlap and should not be summed. An earlier version of the simplified-view text incorrectly implied these parameters “contribute less than 10% of the variance.”
A variance analysis asks a different question: how should output uncertainty be attributed to inputs? Classical Sobol indices require independent inputs; this dependent model needs explicitly defined primitive drivers or a dependence-aware allocation. Neither method alone measures research value.
Status: Association wording corrected; alternative methods now implemented and compared. The source text describes the chart as a “conditional-mean swing statistic, not a variance decomposition.” The sensitivity methods page (September 2026) works through why the bars are hard to interpret — inputs coupled by construction, the arbitrary 10% tails, the mean’s dependence on the right tail, and the difference between learning an input and setting it — and runs the alternatives on the same scenario: adjustable tail widths and output statistics with simulation standard errors, conditional-mean profiles, expected uncertainty after learning one input, rank regression, one-at-a-time interventions, and an exploratory sample-based Shapley-effect estimate. A Sobol decomposition on independent primitive inputs and any value-of-information analysis remain open. The chart itself is still a practical, readable guide to which sliders move the dial most.
Elicitation design issues
Form depth: lightweight survey, not full distribution elicitation
Issue. The focal question asks for a median and optional 80% credible interval. Formal expert elicitation protocols (Sheffield Elicitation Framework, Cooke’s Classical Model) recommend eliciting full probability distributions per quantity, with calibration training, piloting, and revision rounds.
Status: Partially addressed. The Expert Distribution Mode section of the beliefs form asks for p10/median/p90 for five key model parameters (basal media $/L, growth factor price, growth factor dosage, and supplemental protein costs). For richer distributions, Metaculus is linked. A more rigorous follow-up process is planned if interest and resources allow — see Interested in going further? below.
Mixed expert types: subgroup reporting needed
Issue. The workshop mixes TEA authors, industry operators, bioprocess researchers, evaluators, and AW stakeholders. Good-practice elicitation recommends reporting subgroup beliefs separately before any pooled synthesis — subgroup disagreement is information, not noise.
Status: Committed. The beliefs form now uses nine expertise categories (TEA author, bioprocess researcher, industry operator, etc.) enabling subgroup comparison. We commit to reporting subgroup distributions separately in the workshop synthesis.
No formal calibration training
Issue. Formal expert elicitation includes training on cognitive biases in probability estimation — anchoring, overconfidence, availability — with practice questions and feedback. This workshop does not include this: participants are not paid, time is limited, and there is no trained facilitator running a formal protocol.
Status: Noted; future option. The beliefs form asks whether participants have done formal elicitation training before, and whether they’d be interested in a more rigorous process in future.
What the workshop itself is designed to do
The workshop is a deliberation exercise — not primarily an aggregation exercise. The design (pre-workshop beliefs → evidence review → live discussion → post-workshop belief update) is the right structure for collective learning even with a simpler elicitation instrument. The value of having TEA authors, industry operators, and evaluators in structured disagreement is not diminished by the form being a survey.
What the workshop cannot do: generate a calibrated posterior that stands on its own as an authoritative estimate of CM_01. For that, we would need the full formal apparatus described below.
Interested in going further?
If there is enough interest from participants — and if we secure resources — we would design a follow-up exercise with:
- Calibration training — practice questions with known answers, bias feedback
- Full distribution elicitation — chips-and-bins or quartile method for each key parameter
- Individual-before-group — beliefs elicited privately before any group interaction
- Subgroup comparison — separate posteriors for TEA specialists, operators, evaluators
- Compensation — paid time for serious engagement
- Trained facilitator — in structured expert elicitation (e.g., SHELF protocol)
If you’re interested in participating in or helping lead this, please say so in the beliefs form (“About You” section) or email contact@unjournal.org.
This model is part of The Unjournal’s Pivotal Questions initiative. The Unjournal commissions and publicizes expert evaluations of impactful research to improve how evidence informs funding and policy decisions.