Data collection script or program for Linux

I have a need to harvest data from a web site on a weekly basis and need a program to do the work. I currently do it with a program I wrote that runs under windows but I need something more robust in order to run this more often and it must run on Linux (Slackware [url removed, login to view]).

The project is simple: download some html files and then extract the data in them and put them in a standard delimited ASCII file.

The program will simply query a web site to get the html pages that are available. There are about 100 per calendar date and the need is to be able to download the files starting with the current date and going to an end date. This means downloading between 35,000 and 100,000 html files each time the program runs.

The program will begin by getting the html file and then parse the html file and extract the data contained in it. Then get the next one, and so on. The HTML files keep the same format and are easy to extract as they are simple lists. ALL fields must be extracted.

The list have header information, so the header information must be repeated on each record of data created to ensure that the information stays together. For example, it will give the name and then a list of all the clients for that name.

The final output should be sent to the standard output device so that it can be redirected to a file etc. All fields from all html files that are downloaded should be included on each record. The variation between the 3 types of html files in this project are minor so each record line may have about 30 fields, with some used or unused. The field is simply left blank if it is unused.

The extracted data must be in standard ASCII format where each field is delimited by a character to be specified on the command line.


getfiles <start_date> <end_date> <delimiter> > [url removed, login to view]

The default delimiter, if no delimiter is specified on the command line, should be the pipe symbol | (ascii/decimal 124).

I do not need a program with a fancy user interface. It needs to be simple and functional. It will run at the command prompt with parameters just like all standard linux commands and scripts.

The program can be written using linux script language or PHP, either is ok with me.

Attached is a ZIP file containing samples of the html files as well as the web site information.

The provider must submit a final script with all necessary files, and also the source code if any. He must have tested the program and must submit the results for one run between two dates, of a year time period (example: the provider can run the program on february 1st 2009 and put the end date february 1st 2010. The data collected must be submited to show the program works.).

This is a simple project but please only bid if you have done this type of work before and are sure you can deliver the work. I do not want to waste your time or mine.

Payment for this project will be made via escrow only.

Kemahiran: Linux, PHP

Lihat lagi: linux data collection script, data collection script, linux data collection, php data collection script download, data collection script linux, extract data collection linux, extract data running program linux, collection script, Slackware, scripts website data collection, data collection scripts, linux scripts collection, linux data collection scripts, work collection, the source for linux information, standard c types, source code program php, program site web, php program download, php code for calendar, one harvest, linux source code, in line program, html program code, escrow type of payment

Tentang Majikan:
( 25 ulasan ) Montreal, Canada

ID Projek: #378128

11 pekerja bebas membida secara purata $175 untuk pekerjaan ini


We can help in your project, please check PMB to see our related experience.

$250 USD dalam 3 hari
(219 Ulasan)

Hello,Please refer your [login to view URL] you.

$250 USD dalam 3 hari
(49 Ulasan)

Hello, Please look at the PMB. Regards, Sergey

$200 USD dalam sehari
(19 Ulasan)

I've completed already a few such projects on GAF, so I'll be glad to help you too.

$50 USD dalam sehari
(26 Ulasan)

Please check PM. Thanks.

$160 USD dalam 4 hari
(22 Ulasan)

I can do this job for you. See PM for details.

$120 USD dalam 2 hari
(103 Ulasan)

Hi Can be done using web scrapping. Thanks.

$250 USD dalam 0 hari
(6 Ulasan)

pls check p.m.b, thanks.

$80 USD dalam 0 hari
(1 Ulasan)

I can do this job for you

$240 USD dalam 2 hari
(0 Ulasan)

Sounds like the job perl was designed for. (sounds like fun actually)

$180 USD dalam 5 hari
(0 Ulasan)

I'll do it for you

$150 USD dalam 5 hari
(0 Ulasan)