Skip to content
Menu
Finabase Learn

Hedge funds and AI · AI in investing

Using AI chatbots for stock research: how to check what they say

Why AI chatbots state wrong numbers with confidence, how to check an answer against a filing or a quote, prompt habits that help and what never to paste in.

On this page
  1. What a chatbot is actually doing
  2. Stale memory and live data
  3. Worked example: checking one answer
  4. Ask for sources, then open them
  5. Prompt habits that help
  6. What never to paste into a chat
  7. An answer is not advice
  8. Where chatbots fail
  9. How Finabase does it
  10. Frequently asked questions
  11. Summary
  12. Check your understanding

An AI chatbot can summarize a company in seconds, explain a ratio in plain words and suggest questions you had not thought to ask. It can also tell you, in the same calm and fluent voice, that a company earned an amount it never earned. Both behaviors come from the same machinery, and nothing in the answer tells you which one you are reading.

That makes a chatbot a useful research assistant and a poor source. This lesson explains what a large language model is actually doing when it answers, why its numbers can be confidently wrong or out of date, and how to check an answer against a filing or a quote. It ends with prompt habits that make mistakes rarer and easier to catch, the information you should never paste into a chat, and why an answer, however good, is not advice.

What a chatbot is actually doing

A chatbot such as the ones built into search engines, brokerage apps and research sites runs on a large language model (LLM). The model was trained on a very large body of text to do one thing: given the words so far, predict a likely next word, then the next, and so on. Trained at enough scale, that simple task produces answers that read like the work of a well-informed person.

The US National Institute of Standards and Technology (NIST) describes the weakness that comes with it. In its July 2024 profile of generative AI risks, NIST uses the term confabulation for content that is "confidently stated but erroneous or false", which most people call a hallucination. It explains that confabulations are a natural result of how these models work: they produce text that fits the statistical pattern of their training data, and a fluent, plausible sentence is not always a true one. NIST adds two points that matter for research. A model can produce confabulated citations that seem to support its answer, and it can lay out logical steps toward an answer that is itself wrong.

The Financial Stability Oversight Council, the group of US financial regulators chaired by the Treasury Secretary, made the same point in its 2023 annual report. It warned that generative AI can produce output that is wrong but still in the form of a convincing narrative, that checking it may take an expert, and that some models do not give the same answer twice to the same question.

None of this means chatbots are useless. It means their output is a draft that has not been checked, and that the checking is your job.

Stale memory and live data

A model learns from text collected up to a training cutoff. Everything after that date, such as a new quarterly report, a dividend cut, a merger or yesterday's close, is unknown to it unless the application around the model fetches it and passes it in. Some chat products connect the model to live tools such as a quote feed, a search engine or a filings database. Many do not, or use them only for some questions.

The danger is that a model with no live data rarely says so. Asked "What did Apple earn last year?", it may answer with the most recent year it saw in training, stated as if it were current. Markets move on exactly the facts most likely to be missing: the latest earnings, guidance and price.

Where the facts in a chatbot’s answer come from Your question "What did Apple earn last year?" The language model Writes likely next words, learned from text that stops at a cutoff date. Fluent whether or not it knows. Tools the app connects, if any Live quotes, filings, news: only what the app wires in, and only if the model actually calls them. The answer Check each number against the source. A number that came only from the model’s memory may be out of date or invented.
Where an answer’s facts can come from. A figure the model produced from memory may be out of date or invented; one fetched by a tool may be current.

So the first question to ask of any number in an answer is where it came from: the model's memory, or a source the app fetched? If the product cannot tell you, treat the number as unverified.

Worked example: checking one answer

Suppose you ask a chatbot for Apple's latest annual results and it answers: net sales of $391.0 billion, net income of $93.7 billion, diluted earnings per share of $7.46, sales growth of about 8% and a net margin of about 27%. Every figure sounds reasonable. Apple's annual report on Form 10-K for the fiscal year that ended September 27, 2025 settles them in a few minutes; the consolidated statements of operations are on page 29. (How to read a 10-K: a guided walkthrough of Apple’s annual report walks through that report item by item.)

An invented chatbot answer about Apple, checked against the 10-K The chatbot said Apple 10-K, fiscal 2025, page 29 Verdict Net sales $391.0 billion $416.2 billion ($416,161m) Stale: the fiscal 2024 figure Net income $93.7 billion $112.0 billion ($112,010m) Stale: the fiscal 2024 figure Diluted EPS $7.46 $7.46 Matches Sales grew about 8% 6.4% (416,161 ÷ 391,035 − 1) Wrong: no source says 8% Net margin about 27% 26.9% (112,010 ÷ 416,161) Matches, rounded
An invented chatbot answer checked against Apple’s Form 10-K for the fiscal year ended September 27, 2025 (statements of operations, page 29). Finabase calculation.

Three lessons come out of this one check. First, a wrong answer can be mostly right: three of five figures were correct or close, which makes the two stale ones easy to accept. Second, the errors were not random. The model mixed one year's numbers with another's, a typical result when its memory holds both. Third, the answer contradicted itself, which you can catch with arithmetic alone. Net income of $93,736 million divided by EPS of $7.46 implies about 12,565 million shares, while the real net income gives about 15,015 million. When figures in an answer do not fit together, at least one of them is wrong.

Ask for sources, then open them

Asking "Where does that number come from?" is a good habit, but it is not a check by itself. A model can name a filing, a news story or a web page that does not contain the number, or that does not exist. A source counts only when you open it and find the figure there, for the period the answer claims.

Prefer primary sources, and know which one answers which question:

Question Where to check
Revenue, earnings, cash flow, debt The latest 10-K (annual) or 10-Q (quarterly) on SEC EDGAR
A sudden event: merger, CEO exit, deal The company's 8-K and its press release
Today's price, change, volume A live quote, such as a symbol page; note the time it was taken
Dividends and splits The company's announcement and the exchange or a quote page's history
An analyst's or a news outlet's claim The original article, with its date, not the chatbot's summary of it

SEC filings explained: 10-K, 10-Q, 8-K, proxy, 13F and more explains which filing holds what, and How to research a stock: a step-by-step guide for beginners puts the steps in order.

Prompt habits that help

How you ask changes how often a chatbot goes wrong and how easy its mistakes are to spot.

  • Name the company, ticker and period. "Apple (AAPL), fiscal year ended September 27, 2025, as reported in the 10-K" leaves far less room to guess than "Apple's latest results".
  • Ask what the answer is based on. "Is this from your training data or from a live source? What date is your data from?" A product with live tools can often say; one without them may not know.
  • Ask for the calculation, not just the result. "Show the numbers you used to get the growth rate." A shown calculation can be checked line by line; a bare percentage cannot.
  • Invite "I don't know". Ask it to say so when it is unsure rather than to estimate. It will not always comply, but the instruction helps.
  • One question at a time. A long, mixed request produces a long, mixed answer in which a wrong figure hides more easily.
  • Ask for the other side. "What are the strongest arguments against this view?" An assistant tends to go along with the framing of your question, so a leading question gets an agreeable answer.
  • Ask twice, or rephrase. If two answers to the same question disagree, neither is reliable until you check it.

What never to paste into a chat

Treat a chat box like an email to a stranger at a company you do not control. What you type may be stored with your account, reviewed to improve the service, or exposed if the account is breached. NIST's profile lists data privacy, including the leakage or unauthorized disclosure of personal information, among the main risks of generative AI.

Never paste:

  • Passwords, PINs and one-time codes, including two-factor codes. No legitimate assistant needs them, and a request for one is a warning sign (Phishing and smishing attacks on investors: spot them and report them).
  • Account and card numbers, routing numbers and full statements. To ask about a position, type the symbol, share count and cost; leave out the account.
  • Identity details: Social Security number, date of birth, passport or driver's license images, tax forms.
  • Screenshots of brokerage or bank screens without first cropping out names, numbers and balances.
  • Other people's personal information, and anything your employer would treat as confidential, including material nonpublic information about a company.

Be careful in the other direction too. A web page or document you paste in can contain text that tries to steer the model, and a summary of a promotional page can repeat its claims as fact.

An answer is not advice

A chatbot knows nothing about your income, debts, taxes, timeline or tolerance for losses unless you tell it, and you should not tell it the details that would make its advice personal. It owes you no duty. An investment adviser registered with the SEC owes clients a fiduciary duty; a chatbot does not (Financial advisors and fiduciary duty: who must act in your interest).

The SEC, FINRA and the state regulators' association NASAA made the point in a joint investor alert in January 2024: do not rely solely on AI-generated information to make investment decisions. They noted that it may rest on false or outdated news, or on information spread on purpose to move a stock's price, and that even accurate inputs can produce faulty or made-up output. The same alert warns about investment pitches that use the word "AI" to sell guaranteed returns; AI trading bot and signal group scams: the pitches and the tells covers those.

Where chatbots fail

  • Recent events. New filings, earnings, price moves and corporate actions are the facts most likely to be missing or stale.
  • Precise figures. Exact numbers, dates and percentages are where confabulation does the most damage, because a near miss looks right.
  • Small and foreign companies. Less text about them means less for the model to learn from, and more guessing.
  • Definitions that vary. "Earnings" may mean GAAP net income or an adjusted figure; "debt" may or may not include leases. Ask which.
  • Opinions dressed as facts. A confident summary of "the consensus" may be one article, or none.
  • Your own bias. A leading question tends to get an agreeable answer, and an answer that confirms what you hoped is the one least likely to be checked.

How Finabase does it

Finabase has two AI features. On the web, Ask Finabase AI: research stocks with the AI chat on the web (AI Chat in the header) needs a free account. Each answer can come with live data cards, such as a quote card with the price, extended hours, ranges and a five-day chart, drawn from Finabase's own market data, so the figures the answer talks about sit beside it. When an answer draws on recent news, a Sources list under it links each article with its publisher and time, and Retrieved N data sources shows what the assistant looked up, so the source-checking habit above is one click away. You can type / for prompt templates such as /quote, /compare, /signals, /news, /financials and /explain, and $ to mention a symbol exactly, which helps with the "name the ticker" habit above. Signed in, it can read your own watchlists, portfolios and alerts.

Every question counts against a daily limit: free accounts get three requests a day, shared with AI features in the Stock Master app, and the count resets at midnight UTC. Chats are stored with your account; a Temporary chat is not saved, and any chat can be deleted. The chat says plainly that AI can make mistakes and that nothing it says is investment advice, and the guide's advice is the same as this lesson's: verify anything important on the symbol page.

In Stock Master, the AI tab on a quote generates a short written analysis that you can question with follow-ups; each report is labelled as AI-generated, for information only and possibly mistaken (Trend Score and AI analysis in Stock Master). That guide's rule fits any chatbot: numbers in the report should match the quote screen and the company's filings, and if they don't, trust the filings.

Frequently asked questions

Can I trust the numbers an AI chatbot gives me?

Not without checking. A chatbot can state wrong or out-of-date figures with complete confidence. Check every number that affects a decision against the company's filings or a live quote.

Why does a chatbot make up numbers?

Because it generates likely text rather than looking facts up. When it has no reliable figure, it can still produce a plausible one. NIST calls this confabulation and describes it as a natural result of how these models are built.

Is it safe to tell an AI chatbot about my portfolio?

Symbols, share counts and costs are usually enough to ask useful questions. Leave out account numbers, passwords, codes, identity documents and anything else that would hurt you if it leaked.

Can an AI chatbot tell me which stock to buy?

It can produce an answer that sounds like a recommendation, but it does not know your situation and owes you no duty. Use it to understand a company and to find questions to ask, not to decide.

Do chatbots know today's stock price?

Only if the app connects the model to live data and the model uses it. A model working from its training data alone does not know today's price and may not say so.

Summary

A chatbot writes likely text, and likely text is not always true. Its figures can be stale because its memory stops at a cutoff, invented because it fills gaps with plausible numbers, and wrapped in made-up citations. Use it to explain and to suggest questions, then check every number that matters in a filing or a live quote, keep passwords, codes and account details out of the chat, and treat the answer as a starting point, never as advice.

Check your understanding

4 questions on this lesson. Each answer comes with the reason, and nothing is scored.

  1. A chatbot tells you a company's net income for "last year" in a confident, detailed answer. What is the best next step?

    1. Find the figure in the company's latest 10-K or 10-Q and compare it, period by period
    2. Trust it, because a detailed and confident answer is unlikely to be wrong
    3. Ask the chatbot whether it is sure, and accept the figure if it says yes
    Show the answer

    A. Find the figure in the company's latest 10-K or 10-Q and compare it, period by period Confidence and detail are no evidence of accuracy: a language model writes likely text and can state a stale or invented figure in the same voice as a true one. Asking it to confirm only produces more generated text. The filing is the record.

  2. A chatbot gives Apple's net income as $93.7 billion and its diluted EPS as $7.46 for the same year. The 10-K's net income and EPS for that year imply about 15 billion diluted shares. What does the arithmetic tell you?

    1. The two figures do not fit together, so at least one of them is wrong
    2. Nothing: net income and EPS are unrelated numbers
    3. Both are right, because EPS is always rounded
    Show the answer

    A. The two figures do not fit together, so at least one of them is wrong EPS is roughly net income divided by diluted shares. $93.7 billion ÷ $7.46 implies about 12.6 billion shares, far from the roughly 15 billion that the filing's own figures ($112.0 billion ÷ $7.46) imply. In the worked example the net income was the prior year's figure; internal inconsistency is a quick tell even before you open the filing.

  3. The chatbot cites a news article for a revenue number. When does that citation count as support?

    1. When you open the article and it contains that number for the same period
    2. Whenever the chatbot provides a link
    3. When the article comes from a well-known publisher
    Show the answer

    A. When you open the article and it contains that number for the same period Models can produce confabulated citations: a real-looking source that does not contain the figure, or does not exist. A citation supports a claim only after you have read the source and found the number, for the period the answer claims.

  4. Which of these is reasonable to type into an AI chatbot when asking about your holdings?

    1. The symbols, share counts and what you paid
    2. Your brokerage login, so it can read your positions
    3. A screenshot of your statement showing your account number and address
    4. The six-digit code your broker just texted you
    Show the answer

    A. The symbols, share counts and what you paid Symbols, share counts and costs are enough for most questions and reveal little about you. Passwords, one-time codes, account numbers and identity details should never go into a chat: what you type may be stored, reviewed or exposed, and a request for a code is a classic sign of a scam.

Sources and further reading

  1. NIST — AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (July 2024), section 2.2 Confabulation and 2.4 Data Privacy
  2. SEC, NASAA and FINRA — Investor Alert: Artificial Intelligence (AI) and Investment Fraud (January 25, 2024)
  3. Financial Stability Oversight Council — 2023 Annual Report (approved December 14, 2023), section 3.3.3 on AI, pages 90 to 92
  4. Apple Inc. — Form 10-K for the fiscal year ended September 27, 2025, consolidated statements of operations, page 29
  5. Finabase calculations: an invented chatbot answer checked against the Apple 10-K figures above