Archive for October, 2009

Pitch FX mysql database with pitch type

Posted by Darrell in Baseball Data on October 24, 2009

As always please help me with my bandwidth costs for this data. Just buy me a beer. Thanks,

2007-2009 Pitch FX database in mysql

I have went through this year and reworked the import scripts from Mike Fast and especially the mysql database to make the import faster much much much faster with indexes. It was taking over 5 minutes to import a game in 2009 with all the previous data in the database, and now it takes 30 seconds at most. I fixed games that had data errors, and made sure they imported. There is alot of time and effort that goes into this import with my brother bugging me to scripting it, to my brother bugging me, to testing, to my brother bugging me. It really isn’t that bad, but some times it feels like it. The import is up to 151MB compressed so we might have to look at splitting this up by year in the future or something. Ideas? I will only be releasing one file from now on for the Pitch fx MySQL database import is named pbp2.sql. Here is the reworked file that gets updated daily.

Please would also like to know when the 2009 data from retrosheet is out so I can import that data, and maybe rework the output. I have have hear people wanting it out by 10 or 20 years spans. I can do it I just would like to know that people will use it before I do it.

Download Pitch F/X Database here

Here is how it is all done. I have 4 scripts that run a night
1. hack_4day.pl

  • Downloads files from MLB for only the last 4 days to speed up the import.
  • Deletes files older then 4 days.

    • 2. hack_pbp2.pl

  • Downloads all files from MLB to make sure I have a full set of xml files to use if needed.

    • 3. 2009.pl

  • Imports the xml files that are downloaded by the hack_4day.pl script into the pitch fx mysql database.
  • Deletes each games xml files to keep the process clean as to only store one set of xml files and one set of records in the pitch fx mysql database

    • 4. update_db_with_count.pl

  • Updates db with counts on the pitches. great script.

    • All the scripts are available on the downloads page

    22 Comments

    Default server build

    Posted by Darrell in General Information on October 23, 2009

    default server build

    munin-node nagios-nrpe-server nagios-plugins nagios-plugins-basic nagios-plugins-standard ldap-auth-client postfix rdist

    ln -s /bin/bash /usr/local/bin/bash
    ln -s /usr/bin/rdist /usr/local/bin/rdist

    vi /etc/auth-client-config/profile.d/open_ldap
    ###################################################################
    [open_ldap]
    nss_passwd=passwd: files ldap
    nss_group=group: files ldap
    nss_shadow=shadow: files ldap
    nss_netgroup=netgroup: files ldap
    pam_auth=auth required pam_env.so
    auth sufficient pam_unix.so likeauth nullok
    #the following line (containing pam_group.so) must be placed before pam_ldap.so
    #for ldap users to be placed in local groups such as fuse, plugdev, scanner, etc …
    auth required pam_group.so use_first_pass
    auth sufficient pam_ldap.so use_first_pass
    auth required pam_deny.so
    pam_account=account sufficient pam_unix.so
    account sufficient pam_ldap.so
    account required pam_deny.so
    pam_password=password sufficient pam_unix.so nullok md5 shadow
    password sufficient pam_ldap.so use_first_pass
    password required pam_deny.so
    pam_session=session required pam_limits.so
    session required pam_unix.so
    session optional pam_ldap.so
    ##################################################################

    auth-client-config -a -p open_ldap

    No Comments