Labha AI: Building a Small Language Model from Scratch — My Journey into LLM Development

Artificial Intelligence is evolving rapidly, and Large Language Models (LLMs) are transforming how we interact with technology. While using existing models is valuable, I wanted to understand what happens behind the scenes: How are language models designed? How do tokenizers work? How does a model learn from text? Can I build my own language model?

These questions inspired me to start Labha AI, my experimental Small Language Model (SLM) project.

1. What is Labha AI?

Labha AI is an experimental language model that I am developing to understand the fundamentals of Large Language Model architecture, tokenization, training, and text generation.

Rather than simply integrating an existing LLM API, my objective is to explore the core components of a language model and develop a model with my own architecture and training pipeline.

2. Model Architecture

The current Labha AI base model has the following configuration:

Component Configuration
Architecture Transformer-based
Total parameters 1.54 million
Vocabulary size 1,821
Embedding dimension 128
Transformer layers 4
Attention heads 4
Context length 128 tokens

This compact architecture is designed for experimentation and learning rather than competing with large production-scale LLMs.

3. My Training Environment

For model development and experimentation, I have used:

  • Framework: PyTorch
  • Programming language: Python
  • Training environment: Google Colab
  • GPU: NVIDIA Tesla T4
  • Training approach: Custom language-model training experiments

Working with a GPU helped me experiment with model training and understand how computational resources affect the development process.

4. Dataset Preparation

One of the important parts of language-model development is preparing good-quality training data.

During my experiments, I worked on a Wikipedia text dataset.

My data-cleaning report showed:

  • Total articles: 549,976
  • Clean articles: 541,831
  • Duplicate articles: 0
  • Short articles: 8,145
  • Processed dataset size: Approximately 1.983 GB

This process helped me understand the importance of data cleaning, preprocessing, duplicate detection, and dataset quality.

5. Challenges I Faced

Developing a language model is not just about writing a neural network. I encountered several challenges during experimentation:

  • Tokenizer vocabulary and token-generation issues.
  • Repeated tokens during text generation.
  • Training stability and loss optimization.
  • Understanding attention mechanisms and context length.
  • Model checkpoint management and inference.
  • Balancing model size, training data, and available GPU resources.

These challenges became valuable learning opportunities.

6. What I Learned

Through Labha AI, I have been exploring:

  • Transformer architecture
  • Tokenization and vocabulary construction
  • Embeddings and positional information
  • Multi-head self-attention
  • Language-model training
  • Loss functions and optimization
  • Text generation and inference
  • Dataset preprocessing
  • GPU-based model experimentation

The project has helped me move beyond simply using AI models toward understanding how they are built.

7. Future Roadmap

My next development goals include:

  1. Improving tokenizer quality and vocabulary coverage.
  2. Experimenting with larger context windows.
  3. Improving generated-text quality and reducing repetition.
  4. Building a more reliable training and evaluation pipeline.
  5. Exploring instruction fine-tuning.
  6. Developing a simple interface for interacting with Labha AI.
  7. Evaluating the model on structured language-generation tasks.

These are development goals, not features I am claiming are already complete.

8. Why I Started This Project

My long-term goal is to become an AI Engineer with a strong understanding of machine learning, deep learning, and language-model architectures.

Labha AI represents an important step in that journey. It is an opportunity to learn through implementation, experimentation, debugging, and continuous improvement.

I believe the best way to understand AI is not only to use existing models, but also to build, experiment, and learn how they work.

This is just the beginning of Labha AI. More experiments and improvements are ahead!

Let's connect and discuss AI, Small Language Models, Transformers, and LLM development.

LabhaAI #SmallLanguageModels #LLM #ArtificialIntelligence #MachineLearning #DeepLearning #PyTorch #TransformerArchitecture #GenerativeAI #BuildInPublic