Scaper Wiki : Cloud Scraping Platform

-
Recently, I was trying to scrape some data over the web. Initially, I struggled to set up the my scraping environment based on Groovy. Soon enough, I came across Scraper Wiki, a cloud scraping platform. They have support for DOM Parsing and SQLLite database. And, the documentation is simple and well maintained. 
Scraper Page
I quickly choose PHP to scrape the data. Then, I could focus on the problem rather than worrying about setting up the platform. 

 Following is the one of the scripts I have written to extract JSON data into SQL Database. 

 1. Refereed to the existing table in ScrperWiki
2. Constructed the URL to fetch player statistics
3. Used json_decode to decode JSON data
4. Stored data in table using composite primary key (player_id,season)


 <?php
require 'scraperwiki/simple_html_dom.php';
ScraperWiki::attach("sqads", "src");
$playerIds=scraperwiki::select("distinct player_id from src.swdata desc");
foreach($playerIds as $playerid){

$json_content= scraperWiki::scrape("http://..../careerStats/".$playerid["player_id"]."_.....js?_1363343668516=");
//print "JSON".$json_content;
 
if(strpos($json_content,"The page is not found") === FALSE){       

        $json_content=str_replace("onPlayerCareerStats(","",$json_content);
        $json_content=str_replace(");","",$json_content);
        $myPlayerData=null;
        $myPlayerData=json_decode($json_content,true); 
        //var_dump($myPlayerData);
        // PLAYER DATA
        $player_full=$myPlayerData["player"];
        $player_fullName=$player_full["fullName"];
        $player_Nationality=$player_full["nationality"];
        $player_DOB=$player_full["dateOfBirth"];
        $player_id=$player_full["id"];
        //print "Id".$player_id;
        //OVERALL Batting Status
        print $player_fullName.$player_id."\n";
        $overAllRecords=null;
        if(array_key_exists("stats",$myPlayerData)){
        foreach($myPlayerData["stats"] as $stsType)
        {
            if($stsType["matchType"]=="TLPI20"){
                $overAllRecords=$stsType;
             }
        }
        if(!empty($overAllRecords))
        {
        $overAllBatting=null;
        $overAllBowling=null;
        $overAllFielding=null;
        $overAllBatting=$overAllRecords["battingStats"];
        $overAllBowling=$overAllRecords["bowlingStats"];
        $overAllFielding=$overAllRecords["fieldingStats"];
        
        // Prepare Overall Record
        
        $allrecord=array(
                'fullname'=> $player_fullName,
                'nationality'=> $player_Nationality,
                'dateOfBirth'=> $player_DOB,
                'player_id'=> $player_id,
                'season'=>'All',
                'BAT_Mat'=>$overAllBatting ? $overAllBatting["m"] : "-",
                'BAT_Inns'=>$overAllBatting ? $overAllBatting["inns"] : "-",
                'BAT_NO'=>$overAllBatting ? $overAllBatting["no"] : "-", 
                'BAT_Runs'=>$overAllBatting ? $overAllBatting["r"] : "-", 
                'BAT_HS'=>$overAllBatting ? $overAllBatting["hs"] : "-", 
                'BAT_Ave'=>$overAllBatting ? $overAllBatting["a"] : "-",
                'BAT_BF'=>$overAllBatting ? $overAllBatting["b"] : "-",
                'BAT_SR'=>$overAllBatting ? $overAllBatting["sr"] : "-",
                'BAT_100'=>$overAllBatting ? $overAllBatting["100s"] : "-",
                'BAT_50'=>$overAllBatting ? $overAllBatting["50s"] : "-", 
                'BAT_4s'=>$overAllBatting ? $overAllBatting["4s"] : "-", 
                'BAT_6s'=>$overAllBatting ? $overAllBatting["6s"] : "-",
                'BAT_Ct'=>$overAllFielding ? $overAllFielding["c"] : "-", 
                'BAT_St'=> $overAllFielding ? $overAllFielding["s"] : "-",
                'BOWL_Mat'=>$overAllBowling ? $overAllBowling["m"] : "-",
                'BOWL_Inns'=>$overAllBowling ? $overAllBowling["inns"] : "-",
                'BOWL_Balls'=> $overAllBowling ? $overAllBowling["b"] : "-",
                'BOWL_Runs'=>$overAllBowling ? $overAllBowling["r"] : "-", 
                'BOWL_Dots'=>$overAllBowling ? $overAllBowling["d"] : "-", 
                'BOWL_Wkts'=>$overAllBowling ? $overAllBowling["w"] : "-", 
                'BOWL_BBM'=>$overAllBowling && $overAllBowling["bbmr"] !== "-" && $overAllBowling["bbmw"] !== "-" ? $overAllBowling["bbmw"] + "-" +$overAllBowling["bbmr"] : "-",
                'BOWL_Ave'=>$overAllBowling ? $overAllBowling["a"] : "-",
                'BOWL_Econ'=> $overAllBowling ? $overAllBowling["e"] : "-",
                'BOWL_SR'=>$overAllBowling ? $overAllBowling["sr"] : "-", 
                'BOWL_4w'=>$overAllBowling ? $overAllBowling["4w"] : "-", 
                'BOWL_5w'=>$overAllBowling ? $overAllBowling["5w"] : "-");
        
        scraperwiki::save(array('player_id','season'), $allrecord); 
        
        // Prepare Seasonal Data
        
        foreach ($overAllRecords["breakdown"] as $seasonData){
            //print $seasonData["seasonId"]."\n";
            $overAllBatting=$seasonData["battingStats"];
        $overAllBowling=$seasonData["bowlingStats"];
        $overAllFielding=$seasonData["fieldingStats"];
        
        // Prepare Overall Record
        
        $allrecord=array(
                'fullname'=> $player_fullName,
                'nationality'=> $player_Nationality,
                'dateOfBirth'=> $player_DOB,
                'player_id'=> $player_id,
                'season'=>$seasonData["tournamentId"]["name"],
                'BAT_Mat'=>$overAllBatting ? $overAllBatting["m"] : "-",
                'BAT_Inns'=>$overAllBatting ? $overAllBatting["inns"] : "-",
                'BAT_NO'=>$overAllBatting ? $overAllBatting["no"] : "-", 
                'BAT_Runs'=>$overAllBatting ? $overAllBatting["r"] : "-", 
                'BAT_HS'=>$overAllBatting ? $overAllBatting["hs"] : "-", 
                'BAT_Ave'=>$overAllBatting ? $overAllBatting["a"] : "-",
                'BAT_BF'=>$overAllBatting ? $overAllBatting["b"] : "-",
                'BAT_SR'=>$overAllBatting ? $overAllBatting["sr"] : "-",
                'BAT_100'=>$overAllBatting ? $overAllBatting["100s"] : "-",
                'BAT_50'=>$overAllBatting ? $overAllBatting["50s"] : "-", 
                'BAT_4s'=>$overAllBatting ? $overAllBatting["4s"] : "-", 
                'BAT_6s'=>$overAllBatting ? $overAllBatting["6s"] : "-",
                'BAT_Ct'=>$overAllFielding ? $overAllFielding["c"] : "-", 
                'BAT_St'=> $overAllFielding ? $overAllFielding["s"] : "-",
                'BOWL_Mat'=>$overAllBowling ? $overAllBowling["m"] : "-",
                'BOWL_Inns'=>$overAllBowling ? $overAllBowling["inns"] : "-",
                'BOWL_Balls'=> $overAllBowling ? $overAllBowling["b"] : "-",
                'BOWL_Runs'=>$overAllBowling ? $overAllBowling["r"] : "-", 
                'BOWL_Dots'=>$overAllBowling ? $overAllBowling["d"] : "-", 
                'BOWL_Wkts'=>$overAllBowling ? $overAllBowling["w"] : "-", 
                'BOWL_BBM'=>$overAllBowling && $overAllBowling["bbmr"] !== "-" && $overAllBowling["bbmw"] !== "-" ? $overAllBowling["bbmw"] + "-" +$overAllBowling["bbmr"] : "-",
                'BOWL_Ave'=>$overAllBowling ? $overAllBowling["a"] : "-",
                'BOWL_Econ'=> $overAllBowling ? $overAllBowling["e"] : "-",
                'BOWL_SR'=>$overAllBowling ? $overAllBowling["sr"] : "-", 
                'BOWL_4w'=>$overAllBowling ? $overAllBowling["4w"] : "-", 
                'BOWL_5w'=>$overAllBowling ? $overAllBowling["5w"] : "-");
        
        scraperwiki::save(array('player_id','season'), $allrecord); 
        
        }
    }
    }
    else{
        print "Empty STATS";
    }
}
else
{
    print "Empty JSON FOR".$playerid["player_id"]."\n";
}
}

?>

Comments

Post a Comment

Feedback - positive or negative is welcome.

Popular posts from this blog

How to prepare your LOB app for Intune?

-
Image
One of the aspects I struggled with Intune is to wrap an LOB app. In the first cut, it feels like just wrapping the app is enough. But, you need to understand some subtle architectural aspects to get your wrapping right.  App features To get started, let's start with app features.  App uses Azure AD authentication App is a hybrid and uses Cordova (Supports iOS and Android) App communicates with Sharpoint online (via REST) and set up as a Azure AD native application. Thus, conditional access is involved here.  "Send logs via email" - this feature is important as wrapping the application will restrict the email/data sharing capabilities based on the policy  Intune Way You can wrap the application in two ways Using Intune Wrapping Tool Use the Cordova plugin Please visit the link for more details  https://docs.microsoft.com/en-us/intune/deploy-use/decide-how-to-prepare-apps-for-mobile-application-management-with-microsoft-intune Intune wrappi
Read more

Information Architecture - Setup your term store to scale

-
In this post I am going to put together a list of best practices to consider when setting up a term store in Sharepoint Online. As you may already know term store is collection of term groups. And each term group consists of number term sets. Each term set contains a list/hierarchy of terms. Team group gives the flexibility in terms of access. You can assign permissions to users on their term group, rather than giving them access to the entire store.  Always maintain a corporate/general term group which contains common term sets across organization.  Every group/business unit should have a term group. Pin the terms from corporate term group as required into Business unit term groups.  For example, you plan to maintain list of document types in common term group. Thus, there is a term set created. Marketing department want to use this term set. But, they want to add a few more terms. In this case, you should create another term set in Marketing department term group and pin the te
Read more

Generate token signing .CER from ADFS Federation Metadata XML

-
Image
While workging on Force.com SSO, our ADFS team has provided me with federation metadata xml only. As per this link , you also need a token-signing certificate from provider to complete the setup and provide the Force.com XML file to ADFS. Now, the question is how to generate .pem/.cer file out of FederationMetadata.xml file.  Edit FederationMetadata.xml file, and search for  <KeyDescriptor use="signing">. You should find more than one entry. Pick any one of them.      2.  Pick the X500Certificate value and save the text as .der file     3.  openssl x509 -in <(base64 --decode FILE_FROM_STEP2.der) -inform DER -out OUTPUT.pem Use <OUTPUT>.per as Identity Provider Certificate.  Reference:  https://ask.auth0.com/t/how-to-convert-saml-federationmetadata-xml-key-to-pem-or-cer/24
Read more