What is an LLMS.TXT File?
The llms.txt file is a proposal to standardise on using a plain text file that website owners can use to declare how they want large language models (LLMs) to access and use their website content. Similar to robots.txt, it helps control whether AI crawlers can index, store, or use web data for training or answering queries.
It is not yet a web standard, but some AI companies like OpenAI and Google may respect its rules if they are implemented.
LLMS.TXT is a debatable topic in the industry; some say it works, others are bullish about it. Some have even shared the logs proving that LLM bots are crawling this file.
I recently stumbled upon this situation where I had to create it for thousands of pages & the free versions I was finding online didn’t have the option to do it for the list of URLs.
That’s when it occured to me that I can build a Logic based Python Code wherein I can share Screaming Frog CSV that has mapping of URL, Title, Description (Meta Description)
Here the Meta Description can act as context text easily.
And just like that, I was able to create a bulk LLMs.TXT file for the website.
And then it occurred to me, you know what, let’s convert this into a streamlit app for the sake of convenience, where a user can simply import the CSV file & the app generates the LLMS.TXT within seconds & it worked.
Kunjal Chawhan founder of Decode Digital Market, a Digital Marketer by profession, and a Digital Marketing Niche Blogger by passion, here to share my knowledge