Web Scraping from ~20 websites: retrieve simple information from catalogs of biological data

  • Status: Closed
  • Hadiah: $300
  • Penyertaan Diterima: 4
  • Pemenang: rkfcccccc

Ringkasan Peraduan

Dear all,

I am looking for a specialist in data scrapping/python who could help me retrieve information from a few websites, using a list of keywords or a uniform structure of web-pages of these websites.

In brief, there are ~20 websites that contain catalogs of germs and their optimal growth temperatures. I need to extract the name of these germs and the values of their optimal growth temperatures, then merge these data into one .csv file, remove duplicates, and make a few minor changes to this file to make it ready to use by others. This way, we can create a unified and the largest list of germs with experimentally determined growth conditions (instead of having 20 different and difficult-to-use websites). The rest are details.

How our work will be organized:

Step 1
I use this contest to choose a person who can do the work. To win this contest, you need to retrieve names of germs and their growth temperatures from these two databases and show me a random print screen (or any other evidence) of the output files:

Database A:
https://www.bcrc.firdi.org.tw/wwwbcrc/WDCM_RefStrain-tableSearch.jsp

You will see a table of germs (Trametes versicolor, Aspergillus brasiliensis, etc) in which each germ having a link to its unique web page. Here is the page that describes the first germ in the list, Trametes versicolor: https://catalog.bcrc.firdi.org.tw/BcrcContent?bid=36525

Each page of this database is organized uniformly, with germ’s names following the word “Organism:” and the optimal growth temperature value following the word “Growth conditions:”


The expected output file describing this database will be a simple .csv file with two columns:

Organism Temperature (°C)
Trametes versicolor 26
Aspergillus brasiliensis 24
Aspergillus flavus 24
and so on.



Database B:
https://www.sci.muni.cz/ccm/index.html

Here you can see a full catalog where you need to click a certain letter or number, pick a “microorganism” (=germ) to get to a page that contains the germ’s name (first two words on the page), and its optimal growth temperature (the last value on the page).

For instance, just go to the Alphabetical List of Species,
then pick the letter “A”,
and the pick the first germ in the list,
it will bring me to the page that contains the germ’s name “Granulicatella adiacens” and its growth temperature “37 °C”

As in the previous case, the output file for this database will be a simple .csv file with two columns:

Organism Temperature (°C)
Granulicatella adiacens 37
Abiotrophia defectiva 37
etc. etc. etc.


Step 2
When I see that you can deliver, we have a brief Zoom/Skype meeting to discuss the rest of the project. I award you the project and we work for a couple of days to finish everything up before I release the payment and we call it a deal.

Attached is the list of the remaining databases, so that you understand the scale and the complexity of the project. I am expecting to work together with you, brainstorming, suggesting shortcuts and solutions and accepting that some websites will be too hard to tackle.

Kemahiran Disyorkan

Maklum Balas Majikan

“Egor is exceptional and highly prolific! Pleasure working with him and will definitely consider for my future projects.”

Gambar Profil sergeyvmelnikov, United States.

Penyertaan teratas dari peraduan ini

Lihat Lebih Banyak Entri

Papan Penjelasan Umum

  • ashikmohann
    ashikmohann
    • 3 minggu yang lalu

    hi, is the contest closed?

    • 3 minggu yang lalu
  • bsharp101
    bsharp101
    • 3 minggu yang lalu

    Are entries still accepted? Thanks in advance

    • 3 minggu yang lalu
    1. sergeyvmelnikov
      Penganjur Peraduan
      • 3 minggu yang lalu

      I will clarify this point by today's 12 pm UK time as I am still figuring out if I can proceed with entry #3 .

      • 3 minggu yang lalu
    2. bsharp101
      bsharp101
      • 3 minggu yang lalu

      Ok, thanks for clarifying and waiting for your confirmation

      • 3 minggu yang lalu

Bagaimana untuk mulakan dengan peraduan

  • Siarkan peraduan anda

    Siarkan Peraduan Anda Cepat dan mudah

  • Dapatkan berjuta penyertaan

    Dapatkan Bertan-tan Penyertaan Dari serata dunia

  • Anugerahkan penyertaan terbaik

    Anugerahkan penyertaan terbaik Muat turun fail - Mudah!

Siarkan Peraduan Sekarang atau Sertai kami Hari Ini!