Scrape Texts from Marxists.org

I want texts scraped from the Marxist Internet Archive, and saved in an SQL dataset. I want the structure of the data such that each paragraph has its own row, and is associated with the document that it is found in. I want separate columns to represent the metadata. This includes columns for: · Author or speaker, e.g., “Lenin” · Title of the document, e.g., “To His Sister Maria” This is NOT the HTML title tag, but a title of the content on the page itself (usually the first H1 or H2 HTML tag) · The citation information of the source document, e.g., “Lenin Collected Works, Progress Publishers, 1977, Moscow, Volume 37, pages 67-68.” · The date that the text was written. Or, if it is a speech, the date that the speech was delivered, e.g., “October, 1893”. This is often different from the first publication date, or of the publication date of the source document. So there is a need to be able to search for specific phrases within the scraped content (regex). · The date that the text was first published, e.g., “1929”. This is often different from when the text was written, or first published. Again, this can be resolved with regex. · The type of document, e.g., Speech, Letter, Paper. When available it is usually in the first or the last paragraph of each page. · The direct audience. If it is a speech, then this is the assembly that the speech was given to. If it is a letter, then this is who the letter was written to, e.g., “Lenin’s sister Maria.” (You only need to include this if the title, description, or metadata at the top of the text document states the direct audience.) - For any meta value that is NOT available, the column should record “N/A”. - Server response status (404, 500…) in a separate column. The list of domains is provided at the end of this document. For each domain the crawling depth should be 3 levels. The scraping script should log every successful scraped URL so that in case the script has to be restarted, there will be no duplication of content. If you cannot scrape a document because it is not saved as text, e.g., if it is a pdf, then save the metadata of the document to a second dataset (one that is just for these non-text documents). This meta-data could be as simple as name of author/speaker, title of the document, and URL. Skip documents that are not in English. Do not scrape them. Use the language Ruby, Python (Rails or Django as a framework, is required). The preferred scraping engine is scrapy but we are open to suggestions. Our goal is to be able to change the configuration file later if the sites change or if we want to add another domain to the list When the job is completed, save the scripts that you used, and deliver them to me along with the dataset of scraped texts, so that I can replicate your process. Deliverables are source code files and a DB dump as a .SQL file along side with [login to view URL] file that outlines how to configure and run the script independently. Scrape the texts from these locations: Lenin’s “Collected Works”: [login to view URL] Stalin’s works: [login to view URL] Malenkov’s works: [login to view URL] Nikita Khrushchev’s “Writings and Speeches” and “Letters to Kennedy” [login to view URL] Leonid Brezhnev’s “Writings and Statements” [login to view URL] Yuri Andropov’s “Writings and Statements” [login to view URL] Konstantin Chernenko’s “Writings and Statements” (One document) [login to view URL] Mao Zedong’s Works [login to view URL] Hua Guofeng’s Documents: [login to view URL] Deng Xiaoping’s Speech: (One document) [login to view URL]