Zos_Kia

OK that’s a fair observation. Honestly my naive guess would be that they simply do not optimize mainline gpt models for the kind of use case you generally have on Api (tool use, multi-step actions, etc…). They need it to be a perky every day assistant not necessarily a reliable worker. Already on gpt-4 i found it extremely mediocre compared to the Claude models of the same time.

I think that’s a more likely explanation than model collapse which is a really drastic phenomenon. A collapsed model will not just fail tasks at a higher rate, it will spit garbled text and go completely off the rails, which would be way more noticeable. It would also be weird that Claude models keep getting better and better while they’re probably fed roughly the same diet of synthetic data.

There is definitely something happening to that effect. A lot of people have been pushing for sovereignty for quite some time and they have wind in their sails currently.

Ah ! Bien vu j’ai effectivement mal interprété le message

La bulle IA n’a (normalement) pas d’impact sur ce genre de composants d’ancienne génération. Par contre à ta place je m’inquièterais plutôt de savoir si il est vraiment possible de changer ce CPU ? Il me semble que les modèles avec U dans le nom sont soudés directement sur la carte mère. J’espère me tromper mais c’est un point à vérifier.

The switch you mention (from 4th gen to 5th gen GPT) is when they introduced the model router, which created a lot of friction. Basically this will try to answer your question with as cheap a model as possible, so most of the time you won’t be using flagship 5.2 but a 5.2-mini or 5.2-tiny which are seriously dumber. This is done to save money of course, and the only way to guarantee pure 5.2 usage is to go through the API where you pay for every token.

There’s also a ton of affect and personal bias. Humans are notoriously bad at evaluating others intelligence, and this is especially true of chatbots which try to mimic specific personalities that may or may not mesh well with your own. For example, OpenAI’s signature “salesman & bootlicker” personality is grating to me and i consistently think it’s stupider than it is. I’ve even done a bit of double blind evaluation on various cognitive tasks to confirm my impression but the data really didn’t agree with me. It’s smart, roughly as smart as other models of its generation, but it’s just fucking insufferable. It’s like i see Sam Altman’s shit eating grin each time i read a word from ChatGPT, that’s why i stopped using it. That’s a property of me, the human, not GPT, the machine.

Thanks for the measured take, you’re right i am painting with a broad stroke here. And there’s great irony in what you’re describing because if that’s what the blockchain is, then it’s just a piece of legal infrastructure for governments and big corporations. A nice bit of tech, and a fine industry for a few companies to rake in billions a year, but nowhere near the trillion-dollar industry that the investors were pissing themselves about. And their thesis was all fucked so they burnt all that cash on consumer facing gizmos nobody wanted, while they should have been quietly pursuing institutional contracts. Really ridiculous way to waste enormous amounts of money.

Looking at an actual disruptive technology really brings into focus the complete bullshit of cryptocurrency hype.

That’s my feeling too. I do pray every day that the house of cards crumbles, but the tech is really something.

I’m sorry but no, models are definitely not collapsing. They still have a million issues and are subject to a variety of local optima, but they are not collapsing in any way. It is not known whether this can even happen in large models, and if it can it would require months of active effort to generate the toxic data and fine-tune models on that data. Nobody is gonna spend that kind of money to shoot themselves in the foot.

I think all those features are already available and working really well in a high-trust society. In any non-crumbling modern country you already have trust systems embedded in people and institutions, not algorithms, and when they fail you have a court system where another human can disentangle the situation and rule one way or the other. This is a much more desirable state than a low-trust society with algorithmically enforced rules.

Because when you fall in a blockchain edge case, you’re fucked and truly fucked. Nobody can come up and save your ass if someone manages to take advantage of you despite the algorithmic safeguards (which may or may not be well coded themselves). Nobody can help if you die suddenly without handing your crypto keys to a trusted party. This kind of problems, which are trivially solved in the real world, are literally impossible to solve in a blockchain.

Sure, fraud and bad faith can happen in real world institutions, but that’s really a marginal risk, everyday millions of transactions go on without a hitch. And when something fails there’s always a chance of getting your day in court. On average, blockchain “solves” a problem which most people will never encounter in their life. I would imagine that there are interesting applications of the tech in high-stakes boring businesses such as logistics and banking but that’s infrastructure that the end user would never even know exists.

Haha don’t know if i’m ai loving but i’m in my 40s if it helps. I was really using boomers as a slur to refer to dumb suits on LinkedIn, didn’t mean to offend.

Yeah i remember that Ed article ! I don’t think the technical aspects are relevant to the newer generation of models, but yeah of course any attempt to compress inference costs can have side effects : either response quality will degrade for using dumber models, or you’ll have re-inference costs when the dumb model shits its pants. In fact the re-inference can become super costly as dumber models tend to get lost in reasoning loops more easily.

This comparison is really common but i totally disagree with it.

NFTs are a total invention, the poster child of a solution looking for a problem. Nobody, anywhere on earth, ever wondered how they could get their hand on procedural art and speculate on it. The whole metaverse thing was the same way. The pure product of people who are so detached from normalcy that they can’t even begin to fathom what humans tend to like and dislike in the real world.

AI has a million problems which don’t need reiterating. I’m not disputing any of that. It may or may not be a technology that’s viable in the long term (economically and environmentally), i won’t speculate on that. But pre-LLMs there was a huge demand for better natural language processing. For semantically aware programs that are able to generalize and don’t need retraining every 4 days. It was kind of the final frontier for software, the limit between “i can do that in a few sprints” and “i’ll need a bunch of PhDs and 2 years of runway to possibly maybe make something work”.

And i understand that final users only see the dogshit copilot integrations that they never asked for, and which are being pushed to their devices against their will, and becoming a privacy nightmare. And i understand that it’s tiring to hear brain-rotted maximalists constantly making up idiotic predictions about humanity’s future while they let the “groundbreaking tool of the day” riffle through their inbox and bank statements. But it would be a categorical error to believe that LLMs are anywhere as useless as, say, NFTs or the Metaverse.

Yeah that’s also something that you have to train for, i’m not super aware of the technicals but model routing is definitely important to the AI companies. I suspect that’s part of why they can pretend that “inference is profitable” as they are already trying to squeeze it down as much as possible.

What a time to be alive, right?

I hate the “Random industry is cooked” trope. Right now if you go on LinkedIn you’ll find a ton of bros proclaiming the death of Hollywood because Seedance can now generate 3 minutes of incoherent eye-candy with vaguely realistic looking special effects.

It’s like brainrot for boomers, you can see they are getting absolutely hypnotized by the loud noises and bright colors, and totally missing that the main character who was running to the right with blue pants is now walking to the left with red shorts.

I mean what’s the point of morging if you’re gonna do it intermittoucly?

To clarify : model collapse is a hypothetical phenomenon that has only been observed in toy models under extreme circumstances. This is not related in any way to what is happening at OpenAI.

OpenAI made a bunch of choices in their product design which basically boil down to “what if we used a cheaper, dumber model to reply to you once in a while”.

I don’t know about other bands but the bit about iron maiden is really stretched.

I guess if you consider the first lineup to be the one for their first concert in a bar’s basement, alright. But if you take the first album, Dave Murray was already in the band and still is.

Pourtant il y a des gens qui prétendent que ça change leur vie/travail

Ils utilisent des modèles frontière qui ont des performances bien meilleures. Par exemple pour l’écriture c’est effectivement impossible sans une fenêtre de contexte conséquente. Déjà avec 50K tu peux obtenir des trucs pas mal. C’est pas du Tolstoi mais ça se lit et surtout ça reste cohérent, ça respecte les beats narratifs que tu as définis, ça fait pas dériver les personnages vers des versions génériques d’eux mêmes, etc…

Le côté agentique est utile aussi, pas pour faire de “vrais” agents mais parceque ce sont des modèles fine-tunés à bien tolérer les inférences longues et laborieuses. Ils restent dans leur voie, font des checkpoints réguliers pour se réaligner avec le contexte original, etc… En comparaison, les modèles de la génération précédente genre Llama et compagnie sont complètement arrachés. Le modèle va totalement vriller au bout de 5000 tokens et partir à 90° de sa tâche initiale.

En local si tu peux faire passer une quant de GLM-4.5-Air par exemple, tu vas avoir des résultats bien meilleurs. Bon ça va pas être particulièrement rapide par contre…

Cette semaine je réécoutais Death Throes of the Republic de Hardcore History. C’est fascinant de voir tous les parallèles qu’on peut faire avec l’époque actuelle ! Et puis c’est rocambolesque, l’histoire de Caius Marius et Sylla ça ferait une série Netflix. Les épisodes sont passés derrière le paywall depuis un moment mais euh… j’en dispose si ça intéresse quelqu’un. Go DM !

Sinon, Isaac Arthur a sorti un épisode rigolo sur les mathématiques extra-terrestres et j’avoue qu’il prend un angle auquel je n’avais jamais pensé. En gros on se dit que les maths c’est universel parceque si je montre un, deux ou trois objets à un alien il va forcément avoir une construction mentale de ce que 1, 2 et 3 représentent, et il va savoir les manipuler de sorte que 1 + 2 = 3 etc… Mais Isaac présente des contre-hypothèses : et si la vie existait sous forme de nuages de gaz par exemple, de sorte que tout est indénombrable ? Tu aurais une civilisation qui a beaucoup de maths pour désigner des proportions, ou des gradients, mais pas vraiment de concept de nombres entiers. Bref, épisode qui fait réfléchir dans une direction inhabituelle.