I think for $6000 you can run an EPYC setup. But the token/sec is going to be objectively slower than the Macs. What you gain on them is speed. I read this [0] on X earlier today which seems like a good guide on how to get yourself up and running. [0] x.com/i/bookmarks/1884342681590960270?p…...

Running it in a MacBook Pro entirely locally is possible via Ollama. Even running the full model (680B) is possible distributed across multiple M2 ultras, apparently: x.com/awnihannun/status/188141227123634…

Someone also got the full Q8 R1 running on a $6K PC without a GPU on 2x EPYC with 768GB DDR5 RAM running at 6-8 tok/s [1]. Will be interesting to see the value/performance compared to next gen M4 Ultra's (or Extreme?) vs NVIDIA's new DIGITS [2] when they're released. [1] x.com/carrigmat/status/1884244369907278… [2] nvidia.com/en-us/project-digits/

These mini models are NOT the DeepSeek R1. > DeepSeek-R1-Distill models are fine-tuned based on open-source models, using samples generated by DeepSeek-R1. The amount of confusion on the internet because of this seems surprisingly high. DeepSeek R1 has 670B parameters, and it's not easy to run it on local hardware. There are some ways to run it locally, like unsloth.ai/blog/deepseekr1-dynamic which should let you fit the dynamic quant into 160GBs of VRAM, but the quality will suffer. Also MLX attempt on a cluster of Mac Ultras: x.com/awnihannun/status/188141227123634…