fly.io
The article discusses the implementation and utilization of LLama.CPP with Elixir through Rustler, exploring the process of creating a library to interface with LLama.CPP, handling memory safely, and using the model for text prediction. It emphasises the importance of memory management and the advantages of leveraging BEAM’s garbage collection for long-living Rust structures.
Main Points
Introduction
Introduction to using LLama.CPP with Elixir and Rustler, presenting the concept and the initial steps towards integration.
Developing the Library
Detailed walkthrough of creating a new Elixir library to interface with rust-llama-cpp, including setting up dependencies and scaffolding.
Memory Management
Explanation of managing memory in the Rust-Erlang bridge using Rustler’s ResourceArc for safe memory handling.
Model Usage
Illustration of the process to invoke and use the LLama.CPP model within Elixir through Rustler for text prediction.
Conclusion
Conclusion noting the potential of rust-llama-cpp integration in Elixir projects and encouraging contributions.
Insights
Discussion on integrating LLama.CPP with Elixir via Rustler for NIF creation, highlighting memory management.
Creating C/C++ Nif’s in Erlang is kind of a project and you need to be especially careful to not cause memory errors bugs. So I found a Rust Wrapper around LLama.cpp and since I have some experience with Rustler I thought I’d give it a go.
The utility of BEAM's garbage collection in managing memory for long living structures built with Rust.
Because we cannot alter the LLama library directly without vendoring we need to wrap it and do the various implementations that the Rustler ResourceArc type requires as a type. The benefit for us is that the BEAM will handle our memory for us and we only need to give it a handle to clean up when it’s done.