80GB in 4bit. But because it only activates one expert at a time, it can run on ... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		MacsHeadroom on April 17, 2024 \| parent \| context \| favorite \| on: Mixtral 8x22B 80GB in 4bit. But because it only activates one expert at a time, it can run on a fast CPU in reasonable time. So 96GB of DDR4 will do. 96GB of DDR5 is better.

Me1000 on April 17, 2024 [–]

WizardLM-2 8x22b (which was a fine tune of the Mixtral 8x22b base model) at 4bit was only 80GB.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact