Latest Open Source Projects

Building-llama3-from-scratch

TLDR: This repository contains code to build the LLaMA 3 language model from scratch using Python. It explains the components of LLaMA 3 such as pre-normalization using RMSNorm, SwiGLU activation function, Rotary Embeddings (RoPE), and Byte Pair Encoding (BPE) Algorithm. The code shows how to tokenize input data, create embeddings for each token, implement attention heads, self-attention, multi-head attention, SwiGLU activation function, and generate the output.