Open Source AI: Hugging Face, LangChain, and the Ecosystem

Why AI4Bharat matters more than another chatbot demo

AI4Bharat, a research lab housed at IIT Madras, publishes models like IndicTrans2 (translation across 22 scheduled Indian languages) and IndicBERT (a BERT-family encoder pretrained on Indian-language corpora) as public repositories on the Hugging Face Hub. A district hospital building a Kannada symptom-checker, or a fintech building a Bhojpuri voice assistant, does not need to train a language model from scratch — it downloads IndicBERT's weights, fine-tunes a small classification head on a few thousand labelled examples, and ships. This is the concrete difference open source makes in AI: not ideology, but the removal of a capital barrier. Training a competent multilingual encoder from scratch costs lakhs of GPU-hours; downloading one costs a `git clone`.

But "open source AI" is not one thing, and treating it as one thing is the misconception this chapter corrects early, because it changes how you evaluate every model you touch. The stack has three moving parts that are usually taught separately and rarely connected: the Hugging Face Hub and its transformers library, which store and load models; LangChain, which orchestrates models into pipelines that do useful work; and a surrounding ecosystem of formats, licenses, and serving engines that determine whether any of this runs in production at acceptable cost. This chapter builds the mechanism of each, then wires them together in one worked retrieval-augmented generation (RAG) pipeline you can trace by hand.

The Hugging Face Hub: anatomy of a model repository

A model on the Hub is a Git repository with a fixed, boring structure, and that boringness is the entire point — it lets one Python class load ten thousand different architectures without a special case for each. Four files matter:

• config.json — a small JSON file recording the architecture family and hyperparameters: hidden size, number of layers, number of attention heads, vocabulary size, and critically, an "architectures" field naming the exact model class, e.g. ["BertForSequenceClassification"].
• tokenizer.json / vocab.txt / merges.txt — the subword vocabulary and merge rules the tokenizer needs to turn text into integer IDs identically to how the model was trained. A model and a mismatched tokenizer are silently useless: the IDs mean nothing without the exact vocabulary they were fit against.
• model.safetensors — the learned weight tensors. This has replaced the older pickle-based pytorch_model.bin for a serious reason: PyTorch's default checkpoint format uses Python's pickle module, which can execute arbitrary code on load. Downloading a stranger's .bin file and calling torch.load() on it is, in the worst case, running their code on your machine. The safetensors format stores raw tensors with no executable payload, closing that supply-chain hole — a real production concern once you start pulling weights from anonymous Hub uploads rather than a lab you trust.
• README.md — the model card: license, training data description, intended use, and known limitations. It is documentation, but on the Hub it is also machine-readable metadata (license tags, language tags) that search and compliance tooling depend on.

The loading mechanism follows directly from this structure. AutoModel.from_pretrained("ai4bharat/indic-bert") does not guess the architecture — it fetches config.json, reads the "architectures" field, looks up the matching Python class in a registry, instantiates it with the recorded hyperparameters, then streams model.safetensors into that instance's parameters by matching tensor names. AutoTokenizer.from_pretrained does the parallel job for the vocabulary files. This is why the same six lines of code load a 110M-parameter Indic encoder or a 70B-parameter Llama checkpoint: the "Auto" classes are a dispatch layer over config.json, not model-specific code.

Beyond individual repos, the Hub also hosts Datasets (versioned, streamable data repositories with the same Git-based structure) and Spaces (hosted Gradio or Streamlit demos, so a model card can link to a running interface instead of asking you to trust a static description).

LangChain: orchestration, not intelligence

A raw model call answers one prompt with one completion. Almost nothing useful is built that way. A support bot needs to retrieve relevant documents, insert them into a prompt, call the model, parse the output, and maybe call a second tool if the model asks for one. LangChain's job is exactly this glue — it does not make a model smarter; it composes model calls, retrievers, and tools into a directed pipeline, using a small set of composable abstractions:

• PromptTemplate — a parameterized string template that gets filled with runtime values (a question, retrieved context) before being sent to a model.
• Retriever — a uniform interface over a vector store's similarity search, so the rest of the pipeline does not need to know whether the backing store is FAISS, Chroma, or Pinecone.
• Runnable / LCEL (LangChain Expression Language) — the current composition syntax, where pipeline stages are chained with the | operator, mirroring Unix pipes: the output of one stage becomes the input of the next.
• Agent — a loop where the model itself chooses which tool to call next (a calculator, a search API, a SQL query) based on the running conversation, rather than following a fixed sequence.

The most common production pattern built from these pieces is retrieval-augmented generation, introduced by Lewis et al. (2020), "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks," NeurIPS. Instead of relying on facts frozen into a model's weights at training time — which go stale and cannot cite a source — RAG retrieves relevant text at query time and hands it to the model as context, so the model's job shrinks from "know the answer" to "read this passage and answer from it." This is exactly what makes an open-weight 7B model, which knows nothing specific about IRCTC's cancellation policy, capable of answering IRCTC questions correctly: it never has to memorise IRCTC's policy — it just has to read it.

Worked example: tracing a RAG pipeline by hand

Take a toy corpus of three sentences and one query, and trace retrieval exactly — the arithmetic a vector store performs internally, made visible.

Query:  "How can I cancel my train ticket?"
Doc 1 (IRCTC):     "IRCTC allows train ticket booking and cancellation online."
Doc 2 (AI4Bharat):  "AI4Bharat builds open source NLP models for Indian languages."
Doc 3 (UPI):        "UPI enables instant bank transfers using a virtual payment address."

A real embedding model (e.g. sentence-transformers/all-MiniLM-L6-v2) would map each sentence to a 384-dimensional vector. To make the arithmetic checkable by hand, use a toy 4-dimensional stand-in with axes [rail, language, payment, generic], hand-assigned to roughly match each sentence's topic:

query = [0.85, 0.05, 0.05, 0.15]
doc1  = [0.90, 0.00, 0.10, 0.20]   # IRCTC
doc2  = [0.00, 0.90, 0.00, 0.30]   # AI4Bharat
doc3  = [0.10, 0.00, 0.90, 0.20]   # UPI

A vector store ranks documents by cosine similarity: cos(q, d) = (q·d) / (‖q‖‖d‖). Computing doc1 by hand: the dot product is (0.85)(0.90) + (0.05)(0.00) + (0.05)(0.10) + (0.15)(0.20) = 0.765 + 0 + 0.005 + 0.03 = 0.8. The norms are ‖q‖ = √(0.85² + 0.05² + 0.05² + 0.15²) = √0.75 ≈ 0.86603, and ‖doc1‖ = √(0.90² + 0.10² + 0.20²) = √0.86 ≈ 0.92736. So cos(q, doc1) = 0.8 / (0.86603 × 0.92736) = 0.8 / 0.80312 ≈ 0.9961. The same computation for doc2 gives dot = 0.09, norms product ≈ 0.82158, cosine ≈ 0.1095. For doc3: dot = 0.16, norms product ≈ 0.80312 (same as doc1, since doc3 has the same norm), cosine ≈ 0.1992. This code reproduces those numbers exactly:

import numpy as np

def cosine_similarity(a, b):
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

query = np.array([0.85, 0.05, 0.05, 0.15])
doc1  = np.array([0.90, 0.00, 0.10, 0.20])   # IRCTC
doc2  = np.array([0.00, 0.90, 0.00, 0.30])   # AI4Bharat
doc3  = np.array([0.10, 0.00, 0.90, 0.20])   # UPI

for name, doc in [("doc1_irctc", doc1), ("doc2_ai4bharat", doc2), ("doc3_upi", doc3)]:
    print(name, round(cosine_similarity(query, doc), 4))

# doc1_irctc 0.9961
# doc2_ai4bharat 0.1095
# doc3_upi 0.1992

With top_k = 1, the retriever returns only doc1 — correctly, since the query is about cancelling a ticket and doc1 is the IRCTC sentence. In a real LangChain pipeline, the embedding step above is performed by HuggingFaceEmbeddings, the ranking by a FAISS index wrapped as a retriever, and the final answer generation by a Hugging Face model wrapped as a LangChain LLM:

from langchain_huggingface import HuggingFaceEmbeddings, HuggingFacePipeline
from langchain_community.vectorstores import FAISS
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
from transformers import pipeline

corpus = [
    "IRCTC allows train ticket booking and cancellation online.",
    "AI4Bharat builds open source NLP models for Indian languages.",
    "UPI enables instant bank transfers using a virtual payment address.",
]

embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")
vector_store = FAISS.from_texts(corpus, embeddings)
retriever = vector_store.as_retriever(search_kwargs={"k": 1})

generator = pipeline("text-generation", model="google/gemma-2b-it", max_new_tokens=80)
llm = HuggingFacePipeline(pipeline=generator)

prompt = ChatPromptTemplate.from_template(
    "Answer using only the context.\n\nContext: {context}\n\nQuestion: {question}"
)

def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)

rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

print(rag_chain.invoke("How can I cancel my train ticket?"))

The retrieval step in this real pipeline performs exactly the ranking computed by hand above — retriever calls FAISS's cosine search internally and returns doc1 as the sole context; format_docs then joins the retrieved Document objects into the plain sentence text the prompt template expects, rather than letting Python's default list formatting leak into the prompt. What gemma-2b-it actually generates from that context is not something this text can claim without running it — that depends on the model's weights and sampling — but the mechanism up to and including "which sentence gets handed to the model" is fully determined and fully verified by the arithmetic above.

The common misconception: "open source" is not one property

Students (and most journalists) use "open source LLM" to mean "I can download it and it's transparent." In reality there is a spectrum, and conflating points on it leads to real mistakes — for instance, assuming you can inspect a model's training data because its weights are downloadable. Llama 2 (Touvron et al., 2023, Meta AI) ships downloadable weights under the Llama Community License, but Meta has never released the training corpus, and the license itself is not OSI-approved open source — it forbids use by any company with more than 700 million monthly active users without a separate commercial agreement. That makes Llama open-weight, not fully open source: you can run and fine-tune it, but you cannot audit what it was trained on or freely redistribute it under all circumstances. Contrast this with OLMo (Groeneveld et al., 2024, Allen Institute for AI), released with its Dolma training corpus, training code, and intermediate checkpoints all public — a model built explicitly to let researchers study what pretraining data produces what behaviour, not just to let developers call an API-free endpoint.

When evaluating any "open" model for a project, the license file and model card answer three separate questions — can I download the weights, can I see what it was trained on, can I use it commercially at my scale — and the correct instinct is to check all three, not to infer the other two from one.

Production realities: quantization, LoRA, and serving

Downloading a model is the easy 10% of production deployment. Three engineering problems dominate the rest.

Fine-tuning cost. Updating every parameter of a large model (full fine-tuning) requires storing gradients and optimizer state for each of billions of weights — memory that dwarfs the weights themselves. Hu et al. (2021), "LoRA: Low-Rank Adaptation of Large Language Models," observed that the update needed to specialise a pretrained model to a new task is typically low-rank, and proposed freezing the original weight matrix W entirely and learning only a low-rank correction ΔW = B·A, where B is d×r and A is r×k, with rank r chosen far smaller than d or k. For a square d×d attention projection with d = 4096 and rank r = 8, full fine-tuning trains d² = 16,777,216 parameters per matrix; LoRA trains only d·r + r·d = 2dr = 65,536 — a reduction factor of 2r/d = 16/4096 = 1/256, i.e. 256× fewer trainable parameters for that matrix, with the frozen original weights doing the rest of the work unchanged. This is why fine-tuning a 7B open-weight model on a single consumer GPU is feasible at all.

Inference memory. Quantization shrinks a model's weights from 16-bit or 32-bit floats to 8-bit or 4-bit integers (via bitsandbytes, or the GGUF format used by llama.cpp and its wrapper Ollama), cutting memory footprint roughly in proportion — a 7B-parameter model at 16-bit precision needs about 14 GB just for weights, but at 4-bit needs closer to 3.5–4 GB, the difference between requiring a data-centre GPU and running on a laptop.

Serving throughput. Once a model is loaded, serving many concurrent users efficiently is a distinct problem from running one query. Each generated token requires attending back over a growing key-value (KV) cache per active sequence; naive implementations allocate a fixed contiguous buffer per sequence sized for the worst case, wasting most of it for short sequences — internal fragmentation that caps how many requests fit on one GPU at once. Kwon et al. (2023), "Efficient Memory Management for Large Language Model Serving with PagedAttention," SOSP, borrowed the idea of OS-style paged virtual memory: the KV cache is allocated in small fixed-size blocks on demand rather than one large contiguous buffer per sequence, eliminating that waste. Their system, vLLM, reports several-times higher serving throughput than naive Hugging Face transformers generation on the same hardware, purely from this memory-management change — the same underlying model, served smarter.

Diagram: from Hub repository to a running RAG answer

Active recall

Attempt each question before reading its answer.

1. Why can AutoModel.from_pretrained() instantiate the correct architecture without you naming a Python class?
2. State the difference between "open-weight" and "open-source" for an LLM in one sentence, and name one model that is fully open by that stricter definition.
3. The query is changed to emphasise payments: query' = [0.10, 0.05, 0.80, 0.15]. Recompute cosine similarity against doc1 and doc3 and state which document the retriever now returns at top_k=1.
4. A fourth document is added — "IRCTC processes refunds within 3 to 7 working days for cancelled tickets," embedded as doc4 = [0.88, 0.00, 0.15, 0.25] — and top_k is raised from 1 to 2 for the original query [0.85, 0.05, 0.05, 0.15]. Which two documents does the retriever now return, and what changes in the final prompt sent to the LLM?
5. For a d = 4096, r = 8 LoRA adapter on a square attention projection, how many trainable parameters does LoRA use versus full fine-tuning, and what is the reduction factor?
6. What specific memory problem does vLLM's PagedAttention solve, and where does the idea come from?

Worked answers.

1. Because config.json in the repository carries an "architectures" field (e.g. "BertForSequenceClassification") that the Auto class reads and looks up in an internal registry before allocating the model — the class name is data in the repo, not something you supply.

2. Open-weight means the trained parameters are downloadable; open-source (in the fuller sense) additionally means the training data, training code, and license impose no usage restrictions. OLMo (Allen Institute for AI, Groeneveld et al., 2024) is fully open by the stricter definition — weights, the Dolma training corpus, and training code are all public under permissive licenses.

3. dot(q', doc1) = (0.10)(0.90) + (0.05)(0) + (0.80)(0.10) + (0.15)(0.20) = 0.09 + 0 + 0.08 + 0.03 = 0.20; ‖q'‖ = √0.675 ≈ 0.82158; cos(q', doc1) = 0.20 / (0.82158 × 0.92736) ≈ 0.20 / 0.76191 ≈ 0.2625. dot(q', doc3) = (0.10)(0.10) + (0.05)(0) + (0.80)(0.90) + (0.15)(0.20) = 0.01 + 0 + 0.72 + 0.03 = 0.76; cos(q', doc3) = 0.76 / 0.76191 ≈ 0.9975. Since 0.9975 > 0.2625, the retriever now returns doc3 (the UPI sentence) — shifting the query's emphasis toward "payment" flips the top match entirely, exactly as intended.

4. First compute doc4's similarity to the original query: dot(q, doc4) = (0.85)(0.88) + (0.05)(0) + (0.05)(0.15) + (0.15)(0.25) = 0.748 + 0 + 0.0075 + 0.0375 = 0.793; ‖doc4‖ = √(0.88² + 0.15² + 0.25²) = √0.8594 ≈ 0.92704; cos(q, doc4) = 0.793 / (0.86603 × 0.92704) ≈ 0.793 / 0.80284 ≈ 0.9877. Ranking all four: doc1 = 0.9961, doc4 = 0.9877, doc3 = 0.1992, doc2 = 0.1095. At top_k=2 the retriever returns doc1 and doc4 — both IRCTC sentences. The prompt's context field now contains booking/cancellation text and refund-timeline text together, so the LLM can answer a compound question ("how do I cancel and when do I get my refund") that it could not have answered from doc1 alone; doc2 and doc3 remain excluded either way.

5. Full fine-tuning trains d² = 4096² = 16,777,216 parameters for that matrix. LoRA trains 2dr = 2 × 4096 × 8 = 65,536 parameters. The reduction factor is d²/(2dr) = d/(2r) = 4096/16 = 256 — LoRA uses 256× fewer trainable parameters for that matrix while the original weight matrix stays frozen and is reused unchanged at inference time.

6. It solves GPU memory fragmentation in the key-value (KV) cache used during autoregressive generation: naive serving reserves one contiguous buffer per sequence sized for the worst case, wasting memory on every shorter sequence and limiting how many requests fit on one GPU. PagedAttention (Kwon et al., 2023, SOSP) allocates the KV cache in small fixed-size blocks on demand, borrowing the paged virtual-memory idea from operating systems, which removes that waste and lets more sequences run concurrently on the same hardware.

Think About It

Think about this: How would you explain open source ai: hugging face, langchain, and the ecosystem to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where open source ai: hugging face, langchain, and the ecosystem is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting open source ai: hugging face, langchain, and the ecosystem to at least 3 other topics you have studied.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind open source ai: hugging face, langchain, and the ecosystem, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.