Meetup Recap - Aug 24, 2017 August 25, 2017
At the Aug 24 Antradar EEE Toronto meetup, we discussed the following concepts:
- form factor
- execution environment
- use cases / application / the "narrative"
Answers to these questions quickly set the tone of a project.
As an example, the group created a web crawler that indexed all the meeting room locations and phone numbers of the Toronto Public Library.
Form factor: a console script
Execution environment: script run by PHP client program; interfaces MySQL
Narrative: the script downloads the room index, parses out the links, downloads each link, scrapes structured information (library name, geo location, phone number), saves to database.
Near the end of the session we attempted to also build a visualization for the database. For this component:
Form factor: a web page
Execution environment: server-side PHP script provides data; client-side Google Map JavaScript renders the map and markers
Narrative: A map of markers denoting all the Toronto public library meeting rooms.
We distributed a flip diagram toy that captures the flow of a typical web site or application. Only a few toys were available, but more will be made for the next meetup.
source code for crawler.php
<?php
include 'connect.php';
function wget($url){
$curl=curl_init($url);
curl_setopt($curl,CURLOPT_RETURNTRANSFER,1);
curl_setopt($curl,CURLOPT_FOLLOWLOCATION,1);
$res=curl_exec($curl);
return $res;
}
$page=wget('http://www.torontopubliclibrary.ca/using-the-library/room-theatre-rentals/meeting-rooms.jsp');
preg_match_all('/<a href="\/detail\.jsp\?Entt=RDMLIB(\S+?)\&R=(\S+)"><li>([\S\s]+?)<\/li>\s*<\/a>/',$page,$matches);
$libids=$matches[1];
$libnames=$matches[3];
foreach ($libids as $idx=>$libid){
$libname=$libnames[$idx];
$link="http://www.torontopubliclibrary.ca/detail.jsp?Entt=RDMLIB$libid&R=LIB$libid";
$c=wget($link);
preg_match('/<script type="application\/ld\+json">([\S\s]+?)<\/script>/',$c,$matches);
$json=$matches[1];
$obj=json_decode($json,1);
$geo=$obj['geo'];
$lat=$geo['latitude']+0;
$lng=$geo['longitude']+0;
$phone=$obj['telephone'];
$dlibname=addslashes($libname);
echo "$libname $libid $lat $lng $phone\r\n";
$query="insert into libs(extlibid,libname,phone,lat,lng) values ('$libid','$dlibname','$phone',$lat,$lng)";
sql_query($query,$db);
}
We will continue building the map viewer in the next meetup. Stay tuned!
