Python Engineer for Metadata Processing & RAG

@mohjib092

Hi there! I have built large-scale Python ETL and RAG pipelines processing millions of records, including JSONL/GZIP/ZSTD datasets, PDF/EPUB extraction, deduplication, metadata normalization, and vector-ready corpus generation for Qdrant, pgvector, and OpenSearch. I can deliver a production-ready Anna's Archive ingestion pipeline with automated processing, recovery mechanisms, chunking, embedding preparation, and complete deployment documentation. Regards, Muhammad Jibran Ahmed

@lalitdeveloper7

Hi I can build the complete metadata ingestion, deduplication, content extraction, and RAG preparation pipeline for large-scale book and research datasets. I have experience with Python-based ETL workflows, JSONL/GZIP processing, PDF/EPUB text extraction, async data pipelines, vector-search preparation, and scalable data engineering solutions. The pipeline will be designed for reliability, incremental updates, resume/recovery support, and generation of clean embedding-ready JSONL corpora for downstream RAG systems. I have worked on AI/ML platforms involving large dataset processing, document analysis, knowledge extraction, vector databases, and retrieval systems using Python, FastAPI, and modern data engineering practices. The deliverable will include production-ready code, deployment documentation, server setup guidance, and knowledge transfer support. Please let me know further. Thanks.

@Hrolgar

The first thing I'd do is build a blocking layer before any fuzzy comparison: normalize ISBNs (ISBN-10/13 conversion, strip hyphens), then group candidates by normalized ISBN plus publication year. That cuts the comparison space by orders of magnitude before you touch title/author trigram similarity for the ISBN-less records. For cross-database extraction (Postgres/MariaDB/SQLite), I'd use SQLAlchemy with per-source normalization adapters so the dedup logic stays source-agnostic and idempotent. Re-runs process only new records; existing merge decisions are preserved in a resolved_pairs table so you're not re-adjudicating the same pairs on every run. Elasticsearch indexing tuned for the RAG use case: keyword fields on normalized ISBN/author for exact lookup, text with edge-ngram on titles for prefix search, and a dense_vector field stubbed for embeddings if semantic search comes later. The RAG export produces clean JSONL with merge lineage attached so you can trace which source record won each dedup group. Deliverable: dedup pipeline, ES indexer, and RAG dataset exporter, all tested against your actual source schemas. 11,000 INR, 3 days. What does the current schema look like across the three databases: same structure per source, or does each one have its own layout?

@durgesh8527

With over a decade of experience in full-stack development and data engineering, including extensive work in Python, I am confident that I can successfully tackle your ambitious metadata processing project for Anna's Archive. My proficiency extends to managing large-scale datasets with tens of millions of records, designing efficient ETL pipelines, and streamlining workflow processes. Crucial to the project's success is my command over various file formats (JSONL, GZIP, Zstandard), knowledge on deploying async programming (aiohttp, asyncio), and handling text extraction from PDF and EPUBs - all of which I have in ample measure. Additionally, my skills in Linux server administration ensure a reliable and optimized infrastructure setup that aligns with your requirements. Furthermore, having worked extensively on AI-driven projects like LangChain, Claude, OpenAI and Gemini including RAG pipelines and vector databases; I am well-equipped to handle the RAG dataset preparation and embedding tasks as required by your scope of work. You can count on me for not just implementing the pipeline but delivering a clean, production-ready codebase with detailed installation documentation that ensures ease of use even after project completion. My integrated ideology of clean code, clear communication, and on-time delivery ensure I am the perfect fit for your project. Let's connect and build a powerful digital solution for Anna's Archive together!

@Ritesh11gupta

Working with metadata pipelines sounds exciting! I have strong skills in Python and data processing, which I believe would be useful for the book deduplication part of your project. What kind of data volume are you anticipating?

@manoval09

Hi, Book metadata deduplication at scale requires handling fuzzy matches across ISBNs, titles, and author variations—standard string matching falls apart when you're processing thousands of records with inconsistent formatting. I've built similar pipelines for bibliographic data where precision matters as much as performance. I'll architect this around a two-pass deduplication strategy: first pass uses exact-match filtering on normalized ISBNs, second pass applies fuzzy matching (Levenshtein distance with pandas) on title-author pairs, then vectorizes the cleaned metadata into embeddings for your RAG layer. This keeps the pipeline maintainable and lets you tune dedup thresholds without reprocessing everything. In the first 24 hours, I'll map your metadata schema, identify your biggest dedup pain point (ISBN conflicts? author name variations?), and deliver a working prototype on a sample dataset so you see exactly how the pipeline handles your data. What's your primary source of duplicates—multi-edition entries, data imports from different sources, or something else? Best regards, Val

@iamshobhitag

Hi, I've worked on exactly this intersection of large-scale data engineering and LLM/RAG pipelines, and this project maps closely to what I do professionally. Relevant experience • Built production RAG pipelines end-to-end: chunking strategies (token-aware, semantic), JSONL corpus generation, and vector DB ingestion (Qdrant, pgvector). I understand the nuances that kill retrieval quality — chunk overlap, metadata preservation, dedup before embedding. • Processed multi-million record datasets in Python using async I/O (aiohttp, asyncio), Zstandard/GZIP streaming, and incremental ETL patterns. • PDF/EPUB extraction experience using pdfminer, PyMuPDF, and EbookLib. What I'll deliver 1. Metadata ingestion pipeline 2. Deduplication engine — fuzzy title + author normalization, format prioritization (EPUB > PDF > MOBI), cross-source dedup 3. Download + extraction pipeline — async file retrieval, EPUB/PDF text extraction, cleaning and normalization 4. RAG-ready corpus — token-based chunking, metadata enrichment, final JSONL output embedding-ready for Qdrant/pgvector 5. Full docs — deployment guide, server setup, and a knowledge transfer session Portfolio GitHub: Happy to share private RAG project samples over DM. I have availability to start immediately. Happy to jump on a quick call to align on scope before you award. Shobhit

@goranm16

Hi, this fits my work well. Your main goal is not just parsing dumps, but creating a reliable book ingestion pipeline that produces clean, deduplicated, embedding-ready JSONL for RAG. I can help with the Python ETL side: large JSONL/GZ/ZST processing, metadata normalization, deduplication rules, PDF/EPUB text extraction, chunking, logging, resume/retry logic, and storage in PostgreSQL/SQLite or OpenSearch. I’ve worked on document/data pipelines for AI and search-style systems, so I’d focus on clean structure and recoverability from the start. I’d begin with a small end-to-end sample pipeline first, then scale it source by source. The main risk is messy duplicates and failed extraction, so I’d add validation, checkpoints, and clear logs to avoid silent data issues. Assuming all data access is legally permitted, I can deliver this in stages with documentation and handover. Thanks!

@sgwtechnology

Hi — Mukesh here, I can build a production-grade Python data engineering pipeline for large-scale metadata ingestion, deduplication, and RAG-ready dataset preparation. I will design a modular ETL system to process large dumps (JSON/JSONL/GZ/ZST), with async streaming for performance and fault tolerance. The pipeline will include metadata ingestion, author/title normalization, deduplication logic, format prioritization, and version control. It will also handle PDF/EPUB text extraction, cleaning, token-aware chunking, and generate embedding-ready JSONL outputs for vector databases like Qdrant or pgvector. The system will be Linux-ready with logging, resume capability, and scalable architecture for millions of records. Best regards, Mukesh

@souvikbasu

Hi, I'm a Python developer with hands-on experience building data pipelines and RAG systems — I've worked with large-scale document ingestion, vector embeddings, and LLM-powered retrieval architectures similar to what you're describing here. My approach: • Parse Anna's Archive dumps (JSON/JSONL/GZ/ZST) using streaming to handle large file sizes efficiently without memory overload • Build a deduplication layer using ISBN/hash-based fingerprinting across LibGen RS, Fiction, Sci-Hub, and Magazines • Chunk and clean text with configurable strategies (sentence-level, fixed-size, semantic) optimised for embedding quality • Output a clean corpus ready for ingestion into Elasticsearch or any vector DB (Pinecone, Weaviate, pgvector) • Document the full pipeline so you can re-run it as new dumps are released I've built similar pipelines for knowledge base ingestion and can start immediately. Happy to discuss scope and phasing.

@mubashir021

I am excited about the opportunity to work on Anna's Archive Metadata Pipeline and contribute to your AI-powered RAG platform. With extensive experience in Python and large-scale data processing, I can efficiently build the complete ingestion pipeline you need. My background includes developing ETL processes, optimizing metadata workflows, and implementing deduplication engines, ensuring that your datasets are clean and well-structured for embedding and vector database ingestion. I have successfully completed similar projects involving massive datasets, including handling JSONL and GZIP formats. My proficiency in async programming and text extraction from PDF/EPUB files will be instrumental in meeting your project requirements. Additionally, I am well-versed in server setup and configuration, ensuring a smooth deployment and operational monitoring. I estimate a timeline of 14 days to deliver a production-ready codebase, including all necessary documentation and automated scripts. I am committed to maintaining clear communication throughout the project to ensure that your vision is realized effectively. Please check my GitHub profile for relevant projects and my portfolio, showcasing my capabilities in data engineering and pipeline development.

@manishcomp123

**Proposal for Anna’s Archive Metadata Pipeline & RAG Preparation** I’ll build a robust Python-based metadata pipeline to process Anna’s Archive dumps (LibGen, Sci-Hub, etc.) into a clean, deduplicated corpus ready for RAG. My approach includes: 1. **Efficient Ingestion:** Parse large JSON/JSONL/GZ/ZST files with scalable tools (e.g., `ijson`, parallel processing). 2. **Deduplication:** Implement fuzzy matching (e.g., `fuzzywuzzy`, hashing) to identify duplicates across datasets. 3. **Structured Storage:** Design a PostgreSQL schema optimized for metadata and deduplication logic. 4. **Chunking/Prep:** Split text into context-aware chunks (LaTeX/PDF extraction for Sci-Hub) and validate output. I’ve built similar pipelines for academic datasets and can deliver a production-ready system. Let’s align on priorities—should we focus first on deduplication or chunking optimization? Best regards, Manish S.

@Jennycreation

As a seasoned Python developer and data engineer, I bring over 17 years of comprehensive experience in web, window, and Android development. My team has a proven track record of delivering exceptional Python projects, especially in data processing and automation. The range of your project matches perfectly with our expertise. We have hands-on experience with JSONL, GZIP, and Zstandard processing. Our proficiency in asyncio and aiohttp can ensure efficient large-scale data handling, which is crucial for this project. Moreover, we have advanced knowledge in Elasticsearch/OpenSearch and PostgreSQL, essential components for creating an efficient storage schema. Our understanding of ETL pipelines will further benefit the metadata ingestion process. We have previously worked on PDF/EPUB text extraction tasks which are integral to this project as well. thanks

@mohitsps

Estimate: My estimated timeline for this project would be around 3-4 months, depending on the volume of data and complexity of the tasks. As for the cost, I propose $55/hr which is reflective of my years in experience, expertise in large-scale data processing and AI-powered systems. Unique proposition: What sets me apart is my proficiency in text extraction from varied file types and with RAG and vector database systems. This proficiency dovetails with your need for efficient content extraction, metadata enrichment and embedding-ready output all over this pipeline. Speaking of pipeline-strength; no stone will be left un-touched including Linux server setup (Hetzner/VPS), Python environment configuration et al. Conclusion: My technical prowess combined with extraordinary leadership skills ensures that you not only get a completed production-ready codebase but also an installation/deployment documentation and automated ingestion scripts among other things. Don't just gamble with your project on mere claims; my unbeatable record speaks volumes about my ability to align business requirements to technical executions while ensuring predictable and timely delivery - thus making me the best fit for your project!

@parulchoudhary6

Hello, I am an experienced Python Data Engineer and AI Consultant with expertise in large-scale data processing, ETL pipelines, RAG systems, vector databases, and cloud infrastructure. Your project aligns closely with my experience building document ingestion, metadata processing, and AI-ready knowledge pipelines. I can develop the complete workflow, including Anna's Archive metadata ingestion, JSON/JSONL/GZIP/ZST processing, deduplication, author normalization, EPUB/PDF extraction, data cleaning, token-based chunking, metadata enrichment, and generation of embedding-ready JSONL datasets. I also have experience with FastAPI, AsyncIO, PostgreSQL, Elasticsearch/OpenSearch, Qdrant, pgvector, Linux server administration, Docker, and Kubernetes. I have worked on RAG and semantic search solutions involving large document collections, vector search, and LLM-powered retrieval systems. My approach focuses on scalability, fault tolerance, incremental updates, monitoring, and production readiness. I can provide a complete codebase, deployment documentation, automated pipelines, server setup guidance, and knowledge transfer. I would be glad to discuss your expected dataset volume, infrastructure requirements, and preferred vector database to provide an accurate timeline and cost estimate. Looking forward to discussing the project further. Best Regards, Parul Choudhary

@ssquare489

With experience that has taken me from Google to Apple, my journey as a software engineer and developer has instilled in me the understanding that no RAG project can ever truly shine without meticulous data processing at its core. From the parsing of large JSON/JSONL/GZ/ZST files and creation of efficient storage schemas to the handling of incremental updates in Anna's Archive metadata dumps, I have the technological prowess and proficiency to set up your complete book ingestion pipeline. Building on the solid foundation of my Advanced Python knowledge, astute Linux server administration skills, and steep experience with Elasticsearch/OpenSearch, PostgreSQL/SQLite, I will create automated ingestion, deduplication, text extraction and chunking pipelines that culminate in a clean corpus ready for embeddings and vector database ingestion. With LaTex encoding and indexing of PDFs for improved search efficiency under my repertoire, all your Anna's Archive knowledge ingestion needs are aptly covered.

@vyntlabs

Hi! Python data engineering + RAG is my specialty. I've built similar pipelines: metadata parsing, deduplication, and RAG-ready datasets. I can deliver: (1) Robust metadata extraction pipeline, (2) Book deduplication logic, (3) Clean RAG dataset output. Ready to start today. Vynt Labs

@juan99lizano

You need a production-ready pipeline from Anna's Archive metadata dumps through deduplication, text extraction, chunking, and JSONL corpus output ready for vector DB ingestion. I've built a similar ETL pipeline processing 50M+ records from heterogeneous sources — parsing compressed JSONL/ZST dumps with streaming readers, deduplicating via LSH + metadata fingerprinting, extracting text from EPUB/PDF at scale using ebooklib/pdfminer, and outputting token-chunked JSONL for pgvector ingestion. Async aiohttp workers handled file retrieval with resume/retry logic on Hetzner VPS. I can deliver: complete codebase (ingestion → dedup → extraction → chunking → JSONL), Hetzner server setup, deployment docs, and knowledge transfer session. Timeline: 3–4 weeks depending on dataset volume and your target vector DB. Budget expectation: $3,500–$5,000 for full scope. Which vector DB are you targeting — Qdrant, pgvector, or something else — and what's your estimated total corpus size in books?

@ZouhourAbbassi

Hello, I am an AI Engineer and Python Developer with experience building end-to-end AI pipelines, RAG systems, large-scale data processing workflows, and REST APIs. My recent work includes developing AI-powered platforms involving document processing, data cleaning, metadata enrichment, chunking strategies, vector-search preparation, and LLM integration using Python, FastAPI, Hugging Face, and RAG architectures. Relevant experience: • Large-scale dataset processing with Python, Pandas, NumPy, and custom ETL pipelines. • Text extraction, cleaning, and preprocessing for AI applications. • RAG systems, document chunking, metadata enrichment, and embedding-ready datasets. • FastAPI backend development and database integration (MongoDB, MySQL, SQLite). • Linux-based development, Git workflows, and deployment. I have also worked with scientific and research datasets, building production-ready AI workflows and automated data processing pipelines. My background in machine learning and document-based AI systems makes me well-suited for this project. Estimated timeline: 10–14 days (depending on dataset size and infrastructure requirements). I would be happy to discuss the architecture and implementation details. Best regards, Zouhour Abbassi AI Engineer | Python Developer