FlexGen is an open source collaboration among the LMSys community, Stanford, UC Berkeley, TOGETHER, and ETH Zürich.

The high computational and memory requirements of large language model (LLM) inference traditionally make it feasible only with multiple high-end accelerators. FlexGen aims to lower the resource requirements of LLM inference down to a single commodity GPU (e.g., T4, 3090) and allow flexible deployment for various hardware setups. The key technique behind FlexGen is to trade off between latency and throughput by developing techniques to increase the effective batch size.

The key features of FlexGen include:

 High-Throughput, Large-Batch Offloading
Higher-throughput generation than other offloading-based systems (e.g., Hugging Face Accelerate, DeepSpeed Zero-Inference). The key innovation is a new offloading technique that can effectively increase the batch size. This can be useful for batch inference scenarios, such as benchmarking (e.g., HELM) and data wrangling.

 Extreme Compression
Compress both the parameters and attention cache of models, such as OPT-175B, down to 4 bits with negligible accuracy loss.

 Scalability
Comes with a distributed pipeline parallelism runtime to allow scaling if more GPUs are available.

 Limitations
As an offloading-based system running on weak GPUs, FlexGen also has its limitations. The throughput of FlexGen is significantly lower than the case when you have enough powerful GPUs to hold the whole model, especially for small-batch cases. FlexGen is mostly optimized for throughput-oriented batch processing settings (e.g., classifying or extracting information from many documents in batches), on single GPUs.

Learn more by checking out the FlexGen GitHub project and read the supporting paper.

I had to make some updates for jQuery and Bootstrap. “Software” does not age well. Even a lowly app like mine sitting in a GitHub repository needs attention to make sure they work.

Here are some features of the IoT Debugger App:

• ThingSpeak Data Logger and Channel Browser
• Particle.io Webhooks Manager
• Settings are saved in LocalStorage
• Built using HTML5, Bootstrap, and jQuery

I now host a demo of the app using GitHub Pages. Even if you don’t want to download the app and run it yourself, you can just point your browser to https://nothans.github.io/iot-debugger/app/thingspeak.html and use it from there. To use the app, start by entering a channel number. If the channel is public and has data, you should start seeing data on your screen. You can try ThingSpeak channel 1417. This is the channel that stores the CheerLights colors. Use the app to explore your channel data.

Feel free to suggest new features, submit bugs, and make your own changes. That is the awesome thing about open source. We are building this together. See you on GitHub.

I am honored to be on the panel along with Jana Boruta / HashiCorp and Xavier Portilla Edo / Voiceflow.

“As people seek new opportunities to meet and collaborate with like-minded developers locally and globally, community-led open source events are becoming more important than ever. The impact of today’s leading Open Source programs are driven by the grassroots, passionate community organizers.”

Some questions that we will tackle:

• What makes a strong open source community? 
• How can we create authentic experiences? 
• How are leading Open Source professionals measuring the success of their programs? 
• As the world shifts into hybrid and in-person experiences, what does the future of open source community events look like?

Join us Thursday, July 29 at 9:00 AM (PDT) for an expert panel where we’ll discuss these topics, share wins and challenges, and connect with other Open Source community leaders across the world.

Register at Bevy. I am looking forward to answering your questions.

My super simple solution for tracking server disk space is to use Windows PowerShell and ThingSpeak. I went to the trouble of releasing the code to GitHub so that you can try this out for yourself. This can be used on any Windows Server as long as you can execute PowerShell scripts. ThingSpeak gives you a place to store data from anything. In this case, I am sending my disk-free space to ThingSpeak once per day by scheduling a Windows Task.

Check out the open-source code on GitHub!