Meetup Recap - Aug 24, 2017 August 25, 2017

Deer Park Library, Toronto

At the Aug 24 Antradar EEE Toronto meetup, we discussed the following concepts:

  • form factor
  • execution environment
  • use cases / application / the "narrative"

Answers to these questions quickly set the tone of a project.

As an example, the group created a web crawler that indexed all the meeting room locations and phone numbers of the Toronto Public Library.

Form factor: a console script
Execution environment: script run by PHP client program; interfaces MySQL
Narrative: the script downloads the room index, parses out the links, downloads each link, scrapes structured information (library name, geo location, phone number), saves to database.

Near the end of the session we attempted to also build a visualization for the database. For this component:

Form factor: a web page
Execution environment: server-side PHP script provides data; client-side Google Map JavaScript renders the map and markers
Narrative: A map of markers denoting all the Toronto public library meeting rooms.

We distributed a flip diagram toy that captures the flow of a typical web site or application. Only a few toys were available, but more will be made for the next meetup.

source code for crawler.php

<?php

include 'connect.php';

function wget($url){
    $curl=curl_init($url);
    curl_setopt($curl,CURLOPT_RETURNTRANSFER,1);
    curl_setopt($curl,CURLOPT_FOLLOWLOCATION,1);
    $res=curl_exec($curl);   
    return $res;
}


$page=wget('http://www.torontopubliclibrary.ca/using-the-library/room-theatre-rentals/meeting-rooms.jsp');

preg_match_all('/<a href="\/detail\.jsp\?Entt=RDMLIB(\S+?)\&amp;R=(\S+)"><li>([\S\s]+?)<\/li>\s*<\/a>/',$page,$matches);

$libids=$matches[1];
$libnames=$matches[3];

foreach ($libids as $idx=>$libid){
    $libname=$libnames[$idx];
    
    $link="http://www.torontopubliclibrary.ca/detail.jsp?Entt=RDMLIB$libid&R=LIB$libid";
    $c=wget($link);
   
    preg_match('/<script type="application\/ld\+json">([\S\s]+?)<\/script>/',$c,$matches);
    
    $json=$matches[1];
    $obj=json_decode($json,1);
    
    $geo=$obj['geo'];
    $lat=$geo['latitude']+0;
    $lng=$geo['longitude']+0;
    
    $phone=$obj['telephone'];
    
    $dlibname=addslashes($libname);
 
    echo "$libname $libid $lat $lng $phone\r\n";
    
    $query="insert into libs(extlibid,libname,phone,lat,lng) values ('$libid','$dlibname','$phone',$lat,$lng)";   
    sql_query($query,$db);    
      
}

We will continue building the map viewer in the next meetup. Stay tuned!

Our Services

Targeted Crawlers

Crawlers for content extraction, restoration and competitive intelligence gathering.

Learn More

Gyroscope™ ERP Solutions

Fully integrated enterprise solutions for rapid and steady growth.

Learn More

E-Commerce

Self-updating websites with product catalog and payment processing.

Learn More