• SSUPII@sopuli.xyz
    19·
    10 days ago

    If we assume a CD, you can probably fit a 256M parameters model in it. But it will LOAD.

    • yellow [she/her]English
      10·
      10 days ago

      Qwen3-0.6B is about 400 MB at Q4 and is surprisingly coherent for what it is.

      • khepri@lemmy.world
        6·
        10 days ago

        That’s so crazy that an LLM capable of doing anything at all can be that small! That’s leaves room for like an entire .avi episode of family guy at dvd resolution on there, which is the natural choice for the remaining space of course

        • tetris11@feddit.ukEnglish
          2·
          9 days ago

          a 4k episode of family guy using H265 (HEVC) and assuming not too many cutaway gags could produce a file about 240MB. You could probably fit a 480i episode of south park in the remaining 60MB

      • khepri@lemmy.world
        2·
        10 days ago

        Wow, just popped it onto my very slow desktop and this little model rips haha. I really think tiny LLMs with a good LoRA on top are going to be a huge deal going forward

    • lime!@feddit.nu
      3·
      10 days ago

      there’s also tinyllama, which is somewhere around 600MB. it’s hilariously inept. it’s like someone jpeg-compressed a robot.

      also you’re only gonna load off of that cd once so it’ll perform fine.

  • Uriel238 [all pronouns]English
    7·
    10 days ago

    Offline LLMs exist but tend to have a few terabytes of base data just to get started (e.g. before LORAs)

    • nomorebillboards@lemmy.world
      9·
      10 days ago

      I thought it was more like 10-20GB to start out with a usable (but somewhat stupid) model.

      Are you confusing the size of the dataset with the size of the model?

  • MidsizedSedan@lemmy.world
    7·
    10 days ago

    Isn’t it possible to download all of wikipedia, and it being surprisenly a small file size? Can it fit on a CD?

    • yellow [she/her]English
      3·
      10 days ago

      It’s not the LLM that does the web searching, but the software stack around it. On its own, an LLM is just a text completer. What you’d need a frontend like OpenWebUI or Perplexica that would ask the LLM for, say five internet search queries that could return useful information for the prompt, throw those queries into SearxNG, and then pipe the results into the LLM’s context for it to be used.

      As for the models themselves, any decently-sized one that was released fairly recently would work. If you’re looking specifically for open-source rather than open-weight models (meaning that the training data and methodologies were also released rather than just the model weights), GPT-OSS 20B/120B and the OLMo models are recent standouts there. If not, the Qwen3 series are pretty good. (There are other good models out there, this is just what I remember off the top of my head.)