Download of Goblin Spy database
  • Apologies if it's not the correct subforum, I tried to follow the structure of the old forum.

    Is there any way to download the complete Goblin Spy database containing the match data, the player data and the rolls.
    I can view all the info on the website in different windows but I don't find the option to batch download them.

    Also, I heard of a big COL dataset but I can't seem to find it.

  • Might be a thing to consider for the major subsets once a given season is over. Presently goblinSpy divides data into COL seasons and CCL seasons, and then by platform. Each season's data could be turned into a CSV dump and tossed onto a dropbox account or something to save bandwidth, and be linked to from the goblinSpy page.

    The main thing is that so few people work with large data dumps that it has probably never been worth the effort. More people doing stats work with the larger datasets would make it worthwhile, as well as provide the community with expanded insight into what the data is telling us.

    Rolls and player-specific data, on the other hand, are the realm of replay files... I don't think you'll be getting any large dumps of those. Replay files are bloody huge, and for no good reason (they include a full copy of all the match info... every bit of data on the team and the league... with every action... not just every turn.. .every action includes a full copy of it!).. so nobody stores all that many of them for very long.

  • @Loddfafnir
    You can download each query view by using Export but this is of course not a practical solution for getting large amounts of data as I also believe there are some limits in play.

    The match details, rolls and replay shown are extracted from replay files and are not put in any database and I don't plan to do that either. Individual replay files can be downloaded from where you look at the detail match data on goblinSpy.
    It would be possible to fill a DB with replay data, but it would be a lot of data and requests to cyanide so I only ask for it when the user tries to look at the detailed data.

    The database itself is a sqlite database per competition and platform.
    You could join together several of those to get large amounts of data.
    I added a link to the DB to the right of the "menu" on goblinSpy.

    Note that if the DB is updated halfway through your download, the downloaded DB will be corrupt, so best used on non-active competitions.
    You could use http://sqlitebrowser.org/ to browse the data, do queries or export to CSV.

    Tables that correspond to what is given from cyanide are in tables: matches, playerstats, teamstats, coachstats, schedule.

    Tables that correspond to what's shown on goblinSpy are: leaguematches, scheduled, leaguestandings, teammatches, racewins, matchplayers

  • Great, thank you !
    Having a direct access to your SQL databases is a great addition to your project, it will be way easier to do interesting analyses.

    Technical questions : How do you host it ? Is the website on bitbucket / github repo ?

  • @Loddfafnir said in Download of Goblin Spy database:

    Great, thank you !
    Having a direct access to your SQL databases is a great addition to your project, it will be way easier to do interesting analyses.

    Technical questions : How do you host it ? Is the website on bitbucket / github repo ?

    No problemo.
    Web is on a virtual linux hosted by www.glesys.se .
    Backend is c# using HttpListener to expose a web API.
    Static files handled by Apache.
    Source is on https://bitbucket.org/mordrek/goblinspy for those that dare (looking at other peoples code can at times be frightening, frustrating and even revolting at times :) ).

  • 5
    Posts
  • 312
    Views
  • Log in to reply
  • Looks like your connection to Download of Goblin Spy database was lost, please wait while we try to reconnect.