2007-2009 Pitch FX database in mysql

I have went through this year and reworked the import scripts from Mike Fast and especially the mysql database to make the import faster much much much faster with indexes. It was taking over 5 minutes to import a game in 2009 with all the previous data in the database, and now it takes 30 seconds at most. I fixed games that had data errors, and made sure they imported. There is alot of time and effort that goes into this import with my brother bugging me to scripting it, to my brother bugging me, to testing, to my brother bugging me. It really isn’t that bad, but some times it feels like it. The import is up to 151MB compressed so we might have to look at splitting this up by year in the future or something. Ideas? I will only be releasing one file from now on for the Pitch fx MySQL database import is named pbp2.sql. Here is the reworked file that gets updated daily.

Please would also like to know when the 2009 data from retrosheet is out so I can import that data, and maybe rework the output. I have have hear people wanting it out by 10 or 20 years spans. I can do it I just would like to know that people will use it before I do it.

Download Pitch F/X Database here

Here is how it is all done. I have 4 scripts that run a night
1. hack_4day.pl

2. hack_pbp2.pl

3. 2009.pl

4. update_db_with_count.pl

All the scripts are available on the downloads page

munin-node nagios-nrpe-server nagios-plugins nagios-plugins-basic nagios-plugins-standard ldap-auth-client postfix rdist

ln -s /bin/bash /usr/local/bin/bash
ln -s /usr/bin/rdist /usr/local/bin/rdist

vi /etc/auth-client-config/profile.d/open_ldap
###################################################################
[open_ldap]
nss_passwd=passwd: files ldap
nss_group=group: files ldap
nss_shadow=shadow: files ldap
nss_netgroup=netgroup: files ldap
pam_auth=auth required pam_env.so
auth sufficient pam_unix.so likeauth nullok
#the following line (containing pam_group.so) must be placed before pam_ldap.so
#for ldap users to be placed in local groups such as fuse, plugdev, scanner, etc …
auth required pam_group.so use_first_pass
auth sufficient pam_ldap.so use_first_pass
auth required pam_deny.so
pam_account=account sufficient pam_unix.so
account sufficient pam_ldap.so
account required pam_deny.so
pam_password=password sufficient pam_unix.so nullok md5 shadow
password sufficient pam_ldap.so use_first_pass
password required pam_deny.so
pam_session=session required pam_limits.so
session required pam_unix.so
session optional pam_ldap.so
##################################################################

auth-client-config -a -p open_ldap

Getting them installed in Ubuntu is a breeze, thanks to a script I found.
To install the Vista ClearType fonts in Ubuntu, you need to install cabextract first. Cabextract is a utility found in the universe repository, so before you run the following command, make sure you have universe enabled in your repository list. Once this is done, install cabextract using:

Please help with bandwidth costs for this download by donating.

I have imported all the pitch f x data from mlb.com for 2007, to current date using Fastballs wordpress site.  It is available in a mysql export.  Please let me me know if you have any questions, or issues with this download. Please comment and I will get back with you.  If you are looking for game data please check out the retrosheet database I have made available for download  here.

UPDATE 10/23/2009

I have re-imported everything since 2007 up to today, and have it available for download. I will be removing the pbp.sql files, and only going forward with the pbp2.sql which is the pitchfx with pitch type. I should have this done by the time baseball is over. Let me know if you have any questions. I fixed a couple of games that had hit errors where a runner didn’t reach base the system didn’t know what to do with that, so I used a sledge hammer to get rid of the no base runner hit errors. It could also be data import errors but who knows.

Thanks,

Darrell

Converting an ext3 filesystem to ext4

It is possible to mount both ext3 (and ext2, in kernels 2.6.28 and later) filesystems directly using the ext4 filesystem driver. This will allow you to use many of the in-core performance enhancements such as delayed allocation (delalloc) and multi-block allocation (mballoc), and large inodes if your ext3 filesystem have been formatted with large inodes as is the default with newer versions of e2fsprogs. Simply mounting an ext3 (or ext2) filesystem with a modern (2.6.27+) version of ext4 will not change the on-disk structures, and it is possible to revert to the ext3 (or ext2) driver should there be any problem with ext4.

In addition to the in-core performance enhancements, there are additional features which modify the on-disk format from what ext3 understands, such as extents, which can significantly improve the ext4 filesystem performance, but mean the filesystem cannot be mounted by kernels that do not support ext4. There are additional ext4 features, such as flex_bg and > 16TB filesystem support that can only be enabled at format time via mke2fs.

To change an ext2 filesystem (should you still have one) to ext3 (enabling the journal feature), use the command:

# tune2fs -j /dev/DEV

To enable the ext4 features on an existing ext3 filesystem, use the command:

# tune2fs -O extents,uninit_bg,dir_index /dev/DEV

WARNING: Once you run this command, the filesystem will no longer be mountable using the ext3 filesystem!

After running this command, you MUST run fsck to fix up some on-disk structures that tune2fs has modified:

# e2fsck -fD /dev/DEV

NOTE: by enabling the extents feature new files will be created in extents format, but this will not convert existing files to use extents. Non-extent files can be transparently read and written by Ext4.

WARNING: It is NOT recommended to resize the inodes using resize2fs with e2fsprogs 1.41.0 or later, as this is known to corrupt some filesystems.

To upgrade from Ubuntu 8.10 on a desktop system, press Alt+F2 and type in “update-manager -d” (without the quotes) into the command box. Update Manager should open up and tell you: New distribution release ‘9.04′ is available. Click Upgrade and follow the on-screen instructions.

Upgrading from Ubuntu 8.10 Server to Ubuntu 9.04 Server

To upgrade from Ubuntu 8.10 on a server system: install the update-manager-core package if it is not already installed; edit /etc/update-manager/release-upgrades and set Prompt=normal; launch the upgrade tool with the command sudo do-release-upgrade; and follow the on-screen instructions.

1.) Aquire Xten X-lite from Counterpaths website.

$ wget http://counterpath.s3.amazonaws.com/downloads/X-Lite_Install.tar.gz

2.) Untar X-lite_install.tar.gz for use.

$ tar -xzvf X-Lite_Install.tar.gz

This will create a xten-xlite directory with a README file, and the xtensoftphone binary.

3.) Install libstdc++5 which is required to run the xtensoftphone binary.

$ sudo apt-get install libstdc++5

4.) Start the Xten X-lite software on Ubuntu 9.04 and configure it for your phone system.

$ xten-xlite/xtensoftphone

I was not very pleased with this version of Xten X-lite for Linux, but I do like the windows version, and I guess I will be sticking to my twinkle soft phone for now.  When Counterpath gets off it’s butt and builds a newer version of the X-lite for Linux I will try to install it again on my Ubuntu 9.04 Desktop. Please let me know if you have any questions or issues with these instructions and feel free to look at some of my other pages. Nvidia 6200 Xorg.conf with Twinview setup Ubuntu 9.04 Juanty Jackalope or Ubuntu 9.04 BIND DNS Caching server

1.) Install BIND 9 and the BIND 9 utils using apt-get for our BIND DNS caching server.

sudo apt-get update
sudo apt-get install bind9 bind9utils

2.) Point your /etc/resolv.conf to the new servers ip address.

vi /etc/resolv.conf
add “nameserver xxx.xxx.xxx.xxx” to the file

3.) Test your BIND DNS caching server.
dig www.wantlinux.net

You should notice a response the “Query time: 18 msec” from the output of the dig command. Once you make note of the response time repeat the dig command and your response time to decrease drastically. Congratulations your BIND DNS caching server is working. For example my second output was “Query time: 2 msec”. A 16 ms increase in DNS queries might not seem like alot, but when you use spamassassin with DNS options turned on you can make 10-30 DNS queries per email. It adds up quickly when your email server receives 4 messages a second.

Other options for your BIND DNS caching server would be to use your ISP’s dns servers as forwarders, but this does not give you the flexibility of clearing your DNS server cache whenever you want.  If you clear your cache you do not clear your ISP’s DNS cache and you will get the same DNS response. 

Note that BIND caches DNS information to RAM and not disk. In most cases this will not be a problem since most machines have plenty of memory and old records are purged from memory after a period of time. However, if you expect your server to get a lot of traffic you may want to periodically flush the cache using the following command.
sudo rndc -s localhost flush

or set the maximum amount of memory to use (in essence forcing overflow data to be deleted before it expires) by setting the max-cache-size option in the configuration file.

1.) Copy the following lines.
——————————————————————
#!/bin/sh
# Cleanup old rules # All the time firewall is in a secure, closed state
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables –flush # Flush all rules, but keep policies
iptables –delete-chain
## Workstation Minimal firewall ###
iptables -P FORWARD DROP
iptables -P INPUT DROP
iptables -A INPUT -i lo –source 127.0.0.1 –destination 127.0.0.1 -j ACCEPT
iptables -A INPUT -m state –state “ESTABLISHED,RELATED” -j ACCEPT
iptables -A INPUT -p icmp –icmp-type destination-unreachable -j ACCEPT
iptables -A INPUT -p icmp –icmp-type time-exceeded -j ACCEPT
iptables -A INPUT -p icmp –icmp-type echo-request -j ACCEPT
iptables -A INPUT -p icmp –icmp-type echo-reply -j ACCEPT
####### HOLES ####### Edit holes below, then run this script again
# munin
iptables -A INPUT -p TCP -s 192.168.1.16/32 –destination-port 4949 -j ACCEPT
# mysql
iptables -A INPUT -p TCP -s 192.168.1.0.0/24 –destination-port 3306 -j ACCEPT
# nagios
iptables -A INPUT -p TCP -s 192.168.1.16/32 –destination-port 5666 -j ACCEPT
# sshd
iptables -A INPUT -p TCP -s 192.168.1.0/24 –destination-port 22 -j ACCEPT
iptables -A INPUT -p TCP -s 10.0.0.0/24 –destination-port 22 -j ACCEPT
# smtp
iptables -A INPUT -p TCP –dport 25 -j ACCEPT
iptables -I INPUT -p TCP –dport 25 -i eth0 -m state –state NEW -m recent –set
iptables -I INPUT -p TCP –dport 25 -i eth0 -m state –state NEW -m recent –update –seconds 600 –hitcount 45 -j DROP
# pop
iptables -A INPUT -p TCP -s 0/0 –destination-port 110 -j ACCEPT
# imap
iptables -A INPUT -p TCP -s 0/0 –destination-port 143 -j ACCEPT
# 587
iptables -A INPUT -p TCP -s 0/0 –destination-port 587 -j ACCEPT
##################### Edit above
# iptables -A INPUT -j LOG -m limit –limit 40/minute
iptables -A INPUT -j DROP
# Save
# iptables-save > /etc/sysconfig/iptables
echo “: Done.”#!/bin/sh
————————————————————————
2.) Open /etc/init.d/firewall.sh with your favorite editor.

$ sudo vi /etc/init.d/firewall.sh

3.) Paste script into your editor, and edit the ports to your liking. Make sure your are in insert mode in vi before you paste.

4.) Change permissions to all execute on the file.

$ sudo chmod 744 /etc/init.d/firewall.sh

5.) Start the script

$ sudo /etc/init.d/firewall.sh

6.) Add the script to start-up

$ sudo update-rc.d firewall.sh defaults

Please let me know if you have any questions about this really easy and nice Ubuntu iptables firewall script. Like I said I have tried a bunch and this is the best iptables firewall script I have found.