Vibecoding an inference chip

(ankitg.me)

64 points | by ankitvgupta 2 hours ago

8 comments

  • cpldcpu 1 hour ago
    I did something like this with Opus 4.5: https://github.com/cpldcpu/smollm.c

    It seems implementing transformers is the new "hello world" of AI assisted coding.

  • datadrivenangel 2 hours ago
    Does not say what type of model or size they were serving, but one of the traces mentions Maple, which is a 20b model. Still, getting ~70+ tok/s is quite good.
    • ilaksh 13 minutes ago
      Not really. It's only 1B active, and he was probably using the 5 gb quant. Also he said it was over 200 tokens per second.

      But that still not a breakthrough or anything for such a small active set of parameters that is quantized

    • risyachka 1 hour ago
      Fpga is software-driven, model can verify its output easily.

      Zero surprise it did it

    • datadrivenangel 2 hours ago
      it only cost them $10,000 in Claude API usage. :)
      • hmate9 2 hours ago
        You get that much usage in a monthly $200 subscription.
        • dvh 1 hour ago
          So who's paying the cost?
          • stronglikedan 42 minutes ago
            Everyone else that doesn't use the entirety of the plan they pay for.
          • ralph84 49 minutes ago
            API pricing vs. individual plan pricing is just price discrimination. They're both positive gross margin.
          • nyc_data_geek 38 minutes ago
            Retail and society as ANT shares get pulled into index and wealth funds
          • exe34 1 hour ago
            Corporate customers
        • dofm 1 hour ago
          And a kitten.
  • codekansas 1 hour ago
    This is cool

    Next step is tuning your post training pipeline to optimize your model for your chosen FPGA architecture...

  • chalkycrimp 1 hour ago
    Good chance I'm not the only one thinking this: share the code
  • lxe 28 minutes ago
    Repetitive "make it faster" prompting is essentially what all these "RSI" claims are
  • saejox 1 hour ago
    i wonder if we can emulate an FPGA on a rtx 3090? then test it on an AWS EC2 F2 few hours a day.
  • spwa4 42 minutes ago
    I must say I do see one application for this, that you can't really match reasonably otherwise: you can bake weights into an FPGA and then run them for 5W at a speed that will be competitive with a big Nvidia card.

    (unfortunately for a decent model it will also be price competitive with a big nvidia card)

  • Archit3ch 49 minutes ago
    Now do silicon. ;)